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ABSTRACT 


Border  security  is  of  great  importance  to  most  countries.  Turkey  has  been  in 
conflict  with  terrorist  groups  since  the  1980s.  Up  to  now,  more  than  40,000 
people  have  been  killed,  including  Turkish  soldiers  and  civilians.  The  porosity 
and  openness  of  Turkey’s  Iraq  border,  combined  with  the  rugged  topography  of 
the  region,  creates  a  passage  for  terrorist  groups  to  move  materiel  and 
personnel.  Technical  capabilities  of  Unmanned  Aerial  Vehicles  UAVs  can  be 
used  to  improve  coverage  along  borders.  However,  their  effectiveness  is  highly 
dependent  on  the  characteristics  of  the  region.  In  this  study,  87  km  of  the  Turkey- 
Iraq  border  is  modeled  in  Map  Aware  Non  Uniform  Automata  (MANA)  to  examine 
the  potential  impact  of  UAVs  on  detecting  and  classifying  terrorists  seeking 
passage  from  Northern  Iraq  into  Turkey.  The  results  from  the  103,200  simulated 
terrorist  incursions  are  analyzed  using  descriptive  statistics,  stepwise  linear 
regression,  lasso  regression,  regression  trees,  and  random  forests.  The  use  of 
UAVs  is  found  to  be  efficient  in  the  detection  and  classification  of  terrorists  in  this 
region.  The  analysis  techniques  reveal  that  the  most  significant  factors  are  the 
UAV’s  detection  and  classification  performance,  as  well  as  the  terrorists’  counter 
detection  capabilities.  Thus,  Turkey  (and  countries  trying  to  secure  similar 
terrain)  should  purchase  (or  build)  and  employ  hard-to-detect  UAVs  with 
sophisticated  sensors. 
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EXECUTIVE  SUMMARY 


Border  security  is  of  great  importance  to  most  countries.  Many  countries  spend  a 
significant  portion  of  their  budget  protecting  their  border  against  terrorists, 
smugglers,  and  illegal  immigrants.  Illegal  activities  cause  direct  effects,  especially 
for  the  regions  adjacent  to  borders,  and  indirect  effects  for  the  entire  country. 
Turkey  has  been  in  conflict  with  terrorist  groups  since  the  1980s.  Up  to  now, 
more  than  40,000  people  have  been  killed,  including  Turkish  soldiers  and 
civilians.  Terrorist  activities  have  also  ruined  the  socioeconomic  and  social 
stability  of  the  region. 

The  preeminent  terrorist  threat  coming  from  Turkey’s  borders  is  the 
Kurdish  Workers  Party  (PKK).  The  PKK  is  an  armed  Kurdish  organization 
struggling  with  Turkey.  The  organization  is  also  listed  as  a  foreign  terrorist 
organization  by  the  U.S  Department  of  Defense  (DoD).  The  southeast  (SE) 
border  of  Turkey  contains  the  majority  of  terrorist  activities.  The  region  is  known 
for  its  steep  mountains  and  deep  dales.  Thus,  the  soldiers  protecting  the  border 
have  limited  lines-of-sight  (LOS).  These  conditions  reduce  the  probability  of 
detecting  PKK  militants  operating  or  transiting  in  the  region. 

The  PKK  terrorist  organization  also  takes  advantage  of  the  gaps  along  the 
border  in  its  illegal  drug  trafficking,  which  supports  the  organization  financially. 
Terrorists  use  ever  changing  guerilla  tactics.  They  cross  over  the  border  to  attack 
military  outposts  and  to  bring  explosives  for  use  in  bomb  attacks  in  urban  areas. 
Simultaneous  raids  are  a  recently  used  tactic  of  the  terrorist  organization.  In 
addition  to  military  stations,  the  terrorists  attack  military  and  police  lodgings, 
prefectures,  and  government  agencies. 

In  Turkey,  conventional  Turkish  military  and  police  forces  play  a  key  role  in 
both  the  interdiction  of  illegal  drugs  and  in  defending  the  country  against  terrorist 
attacks.  The  porosity  and  openness  of  Turkey’s  Iraq  border  have  been  a  problem 
in  counterterrorism.  The  Qandil  Mountains,  which  are  located  south  of  the 


Turkey-lraq  border,  provide  an  operations  center  as  well  as  tactical  and  practical 
advantages  for  the  terrorists.  Combined  with  the  rugged  topography  of  the 
region,  border  porosity  creates  a  passage  for  terrorist  groups  to  move  material 
and  personnel.  The  terrain  also  makes  it  difficult  for  military  outposts  to  support 
each  other.  In  order  to  decrease  illegal  activities,  including  terrorism,  effective 
border  monitoring  is  essential.  Detection  of  terrorist  threats  is  vital  for  the  safety 
of  the  border  region,  assets  along  the  borders,  and  security  of  the  whole  country. 

Unmanned  Aerial  Vehicles  (UAVs),  aerial  vehicles  that  can  be  piloted 
remotely,  are  a  critical  component  of  modern  day  intelligence,  reconnaissance, 
and  surveillance  (ISR).  Technical  capabilities  of  UAVs  can  be  used  to  improve 
coverage  along  borders.  Advanced  Electro-Optical  (E-O)  identification 
technology,  UAV  loiter  capability,  the  range  of  UAVs  compared  to  other  patrol 
assets,  and  their  low  price  explains  why  UAVs  should  play  a  key  role  in  ISR 
missions.  The  growing  demand  for  UAVs  is  also  based  on  avoiding  military 
casualties.  In  terms  of  today’s  casualty  averse  military  environment,  the 
consequences  of  losing  an  airman  and  a  UAV  cannot  be  compared.  UAVs  can 
be  designed  to  fly  under  extreme  conditions  that  are  risky  for  pilots.  They  can  fly 
over  extreme  altitudes  for  extended  periods  without  suffering  the  emotional  and 
physical  effects  experienced  by  humans.  UAVs  also  improve  situational 
awareness.  However,  their  effectiveness  is  highly  dependent  on  the 
characteristics  of  the  region  in  which  they  are  deployed.  This  thesis  examines  the 
effectiveness  of  UAVs  in  helping  secure  a  border  characterized  by  rough  terrain 
and  active  terrorists. 

The  baseline  incursion  scenario  analyzed  in  this  research  was  developed, 
based  on  a  skirmish  between  a  guerilla  team  and  a  battalion  in  the  Southeast 
region  of  Turkey,  using  Map  Aware  Non  Uniform  Automata  (MANA).  For  this 
thesis,  87  km  of  the  Turkey-lraq  border  was  chosen  as  the  area  of  interest  to 
model  five  different  scenarios  including  different  numbers  of  UAVs  (0  up  to  4). 
The  following  image  provides  an  overview  to  the  battlefield  with  the  agents  for 
the  scenario  with  one  UAV. 

xviii 


4  MAUA  ■  Sceftano.Two.  1 AOR 


Pause 


Red  Agents 


£il«  Soup  Qnplr,  Kww  0  «»  Qutputi  Belp 


ue 

attalions 

F 

Civilians  ^ 

k 

f 

♦ 

'  ti  \ 


I  Scouts 


Blue  killing 
,  -Agent 


RNW 


V  'fto**! 


Blue  (  0 


0 


^MurtiKun 

1-4  Reset 

Step  Delaytmi) 
0  t 

Mex.  Siep* 

9000  S 


Seed 


■  Tttian  Type  Road  addad 

I  Tw«n  Ty^Moi^sr.  added 
It  T«nar>  Type  OftRoad  added 

■  T  ft  ran  T  ,**■  BcfdwO**  added 


C*Uq  Saweloq 


/  CakUareLOS 
J  Irtfia  Squad  Nat 
J  Iraa*  Squad  Nat 


In*  Tae  Dead 
Pdfh 


Model  Step:  0 


NeuCtal  Cat:  0 


£lev*ttonr  1.728  m 


Overview  of  scenario  including  one  UAV. 


UAVs  are  responsible  for  monitoring  the  87  km  border,  which  is  divided 

into  Areas  of  Responsibility  (AORs)  (from  0  up  to  4).  Each  UAV  is  capable  of 

detecting,  classifying,  and  tracking  illegal  entrants.  A  UAV  follows  predefined 

waypoints  in  the  default  state.  Once  it  detects  an  activity,  it  then  proceeds  to  the 

detection  area  for  classification.  The  UAV  tracks  the  classified  agent  if  it  is  a  Red 

Agent.  Otherwise,  the  UAV  keeps  following  its  waypoints.  There  are  two  Blue 

Battalions  in  the  model  and  the  destination  of  Red  Agents  is  set  depending  on 

the  location  of  these  battalions.  Red  Teams  represent  the  terrorists.  There  are 

four  groups  of  Red  Teams  trying  to  cross  the  border.  Initially,  each  group  is 

located  at  four  different  points  along  the  Iraq  part  of  the  border.  They  follow  their 

waypoints  into  Turkey  and  try  to  avoid  the  UAVs  along  their  path.  There  are  17 

Scouts  distributed  along  the  border  to  provide  information  on  UAV  activities  to 

the  Red  Teams.  Scouts  have  a  stationary  observation  point.  Scouts  are  able  to 
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extend  their  sensor  range  by  using  binoculars.  Neutrals  are  the  agents  that 
cause  distraction  to  the  UAVs  until  they  are  classified  as  neutral.  However, 
Neutrals  can  provide  information  on  UAV  activities  to  the  Red  Agents  in  some 
scenarios. 

Following  the  creation  of  the  base  model,  a  Nearly  Orthogonal  Latin 
Hypercube  (NOLH)  design  is  used  to  provide  maximum  information  from  the 
experimental  design  for  the  21  factors  in  this  study.  The  impact  of  UAVs  on 
detection  and  classification  of  terrorists  is  examined  by  varying  factors  in  the 
model:  such  as  the  number  of  UAVs,  UAV  sensor  range,  UAV  speed,  number  of 
terrorists,  detection,  classification  and  communication  range  of  terrorist  and 
scouts,  etc.  In  addition  to  the  factors  varied  in  the  design,  there  are  also  13 
dependent  input  variables  that  are  set  as  a  function  of  some  factors:  UAV 
classification  range,  UAV  classification  probability,  time  between  detections,  and 
UAV  fuel  usage  rate.  The  resulting  experimental  design  is  crossed  with  the 
number  of  UAVs  (1  up  to  4).  Afterwards,  each  of  the  512  design  points  is 
replicated  200  times,  producing  a  final  data  set  of  103,200  observations. 

Descriptive  statistics,  stepwise  linear  regression,  lasso  regression, 
regression  trees,  and  random  forests  are  used  to  analyze  103,200  simulated 
terrorist  incursions.  The  analysis  techniques  highlight  similar  factors  as  the  most 
important  factors,  plus  with  additional  insights.  The  following  list  summarizes  the 
primary  findings  of  our  analysis  conducted  based  on  the  results  of  scenarios: 

•  Agent-based  models  provide  us  a  modeling  platform  to  create  and 
analyze  scenarios  in  a  short  time  period. 

•  The  data  farming  process  is  a  powerful  technique  to  study  the 
effects  of  numerous  factors  simultaneously. 

•  The  use  of  UAVs  significantly  enhances  the  detection  and 
classification  of  terrorists  operating  or  transiting  in  the  region. 

•  The  classification  range  of  a  UAV  has  a  strong  positive  affect  in  the 
number  of  classified  terrorists.  The  first  split  in  the  regression  tree 
analysis  highlights  the  importance  of  having  a  classification  range 
greater  than  5  km. 
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The  sensor  range  of  terrorists  to  detect  UAVs  has  a  negative 
impact  on  the  number  of  classified  terrorists.  Even  though  UAV 
altitude  is  not  highlighted  as  a  significant  factor,  assigning  UAVs  at 
higher  altitudes  can  reduce  their  probability  of  detection  by 
terrorists. 

The  classification  probability  of  a  UAV  is  also  important.  A  1% 
increase  in  classification  probability  results  in  more  classified 
terrorists.  Additionally,  regression  tree  analysis  yields  that  a 
maximum  range  (>  5000  meters)  classification  probability  greater 
than  0.17  provides  better  results  in  the  classification  of  terrorists. 
From  a  practical  sense,  classification  probability  of  a  camera,  which 
is  onboard  the  UAV,  leads  to  the  number  of  effective  looks  needed 
to  search  an  area.  So,  this  is  an  important  factor  to  take  into 
account  in  terms  of  mission  plan  and  organization. 

We  cannot  talk  about  an  optimal  number  of  UAVs  to  assign  over 
the  area  of  interest,  but  regression  tree  analysis  indicates  that 
assigning  three  or  more  UAVs  results  in  more  classified  terrorists 
with  a  lower  variance. 

The  time  between  detections  of  entities  is  another  important  factor 
in  terms  of  a  UAV’s  classification  capability.  A  one  second 
increment  of  time  between  detections  in  the  default  state  causes  a 
larger  decrease  in  the  number  of  classified  terrorists  than  a  one 
second  increment  of  time  between  detections  in  enemy  contact 
state. 

Refueling  time  is  a  bigger  determinant  of  success  than  the  initial 
fuel  level  of  a  UAV.  Thus,  it  is  important  for  ground  bases  to  be  able 
to  quickly  get  the  UAVs  back  up. 

Increasing  the  UAV’s  speed  in  the  default  state  has  a  negative 
impact  on  UAV  classification  performance.  Regression  tree 
analysis  shows  that  UAV  speeds  higher  than  250  km/hr  noticeably 
decreases  the  number  of  classified  terrorists.  Therefore,  the  speed 
of  flight  is  critical.  If  it  is  too  high,  UAVs  will  quickly  occupy  images 
of  the  terrain  and  will  miss  the  details.  On  the  contrary,  if  the  speed 
is  too  low,  then  there  will  be  less  variation  in  the  occupied  image, 
hence  less  information  will  be  gathered  through  the  scans. 

The  existence  of  communication  links  between  the  civilians  and 
terrorists  is  another  significant  factor  on  UAV  performance.  The 
existence  of  civilian  communication  with  terrorists  caused  a 
reduction  in  the  number  of  classified  terrorists.  Therefore,  SIGINT 
and  electronic  warfare  (EW)  capabilities  should  be  considered  in 
addition  to  UAV  performance  characteristics. 
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I.  INTRODUCTION 


“The  future  is  in  the  skies.  ” 

-  Mustafa  Kemal  Ataturk 


A.  BACKGROUND 

Border  security  is  of  great  importance  to  most  countries.  Many  countries 
spend  a  significant  portion  of  their  budget  protecting  their  border  against 
terrorists,  smugglers,  and  illegal  immigrants.  Illegal  activities  cause  direct  effects, 
especially  for  the  regions  adjacent  to  borders,  and  indirect  effects  for  the  entire 
country.  Turkey  has  been  in  conflict  with  terrorist  groups  since  the  1980s.  Up  to 
now,  more  than  40,000  people  have  been  killed,  including  Turkish  soldiers  and 
civilians  [1],  Terrorist  activities  have  also  ruined  the  socioeconomic  and  social 
stability  of  the  region. 

1 .  Overview  of  Turkey’s  Borders 

Turkey  is  a  Eurasian  country  located  in  Western  Asia  and  Southeastern 
Europe.  The  country  has  a  total  of  2,627  kilometers  of  land  boundaries  and  7,200 
kilometers  of  coastline.  Turkey  is  bordered  by  eight  countries:  Bulgaria,  Greece, 
Syria,  Armenia,  Azerbaijan,  Iran,  Georgia,  and  Iraq  [2],  A  majority  of  the  terrorist 
infiltrations  take  place  from  Iraq.  In  Turkey,  conventional  Turkish  military  and 
police  forces  play  a  key  role  in  both  the  interdiction  of  illegal  drugs  and  in 
defending  the  country  against  terrorist  attacks. 

The  porosity  and  openness  of  Turkey’s  Iraq  border  have  been  a  problem 
in  counterterrorism.  The  Qandil  Mountains,  which  are  located  south  of  the 
Turkey-lraq  border,  provide  an  operations  center  as  well  as  tactical  and  practical 
advantages  for  the  terrorists  [3],  Combined  with  the  rugged  topography  of  the 
region,  border  porosity  creates  a  passage  for  terrorist  groups  to  move  material 
and  personnel.  The  terrain  also  makes  it  difficult  for  military  outposts  to  support 
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each  other.  In  order  to  decrease  illegal  activities,  including  terrorism,  effective 
border  monitoring  is  essential. 

2.  Overview  of  Terrorist  Activities 

The  preeminent  terrorist  threat  coming  from  Turkey’s  borders  is  the 
Kurdish  Workers  Party  (PKK).  The  PKK  is  an  armed  Kurdish  organization 
struggling  with  Turkey.  The  organization  is  also  listed  as  a  foreign  terrorist 
organization  by  the  U.S.  Department  of  Defense  (DoD)  [4],  The  southeast  (SE) 
border  of  Turkey  contains  the  majority  of  terrorist  activities.  The  region  is  known 
for  its  steep  mountains  and  deep  dales.  Thus,  the  soldiers  protecting  the  border 
have  limited  lines-of-sight  (LOS).  These  conditions  reduce  the  probability  of 
detecting  PKK  militants  operating  or  transiting  in  the  region. 

PKK  groups  are  well  adapted  to  hard  conditions  and  the  environment. 
Most  of  their  plans  take  advantage  of  the  geography  of  the  region.  The  bridges 
are  the  only  connection  between  some  regions  and  Turkish  battalions.  This  well- 
known  weakness  can  be  used  to  increase  the  response  time  by  keeping 
reinforcement  teams  away  from  an  attacked  Turkish  asset. 

The  PKK  terrorist  organization  also  takes  advantage  of  the  gaps  along  the 
border  in  its  illegal  drug  trafficking,  which  supports  the  organization  financially. 
Terrorists  use  ever  changing  guerilla  tactics.  They  cross  over  the  border  to  attack 
military  outposts  and  to  bring  explosives  for  use  in  bomb  attacks  in  urban  areas. 
Simultaneous  raids  are  a  recently  used  tactic  of  the  terrorist  organization.  In 
addition  to  military  stations,  the  terrorists  attack  military  and  police  lodgings, 
prefectures,  and  government  agencies.  News  from  the  world  press  reflects  how 
terrorist  attacks  pose  a  severe  threat  to  the  safety  of  the  Turkish  people: 

PKK  rebels  launched  a  series  of  attacks  against  security  forces  and 
village  guardsmen  within  three  Turkish  provinces;  §irnak,  Siirt, 
Antakya.  [5] 

PKK  militants  clashed  with  Turkish  soldiers  near  the  town  of 
Cukurca,  within  the  Hakkari  Province,  in  the  south-east  of  the 
country.  [6] 
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A  bomb  explosion  occurred  next  to  a  military  bus  carrying  Turkish 
soldiers  and  their  relatives  in  Turkey’s  largest  city  of  Istanbul.  PKK 
militants  are  being  suspected  for  carrying  out  this  latest  attack.  [7] 

Around  100  suspected  PKK  fighters  simultaneously  attacked  four 
government  and  security  buildings  in  the  small  town  of 
Beytu§§ebap,  near  the  border  with  Syria.  At  least  10  soldiers  and  3 
attackers  were  killed  during  the  assault,  while  7  soldiers  were 
injured.  [8] 

A  roadside  bombing  in  Turkey’s  southeastern  Bingol  Province  killed 
8  soldiers  and  injured  9  others,  less  than  a  day  after  4  officers  were 
killed  in  an  attack  near  the  borders  with  Iran  and  Iraq.  [9]  [10] 

An  explosive  device  hidden  in  a  car  exploded  as  an  Army  patrol 
was  passing  by  in  the  eastern  Turkish  city  of  Tunceli,  killing  6 
soldiers  and  a  civilian.  Several  others  were  injured  in  the  blast, 
which  authorities  blamed  on  the  PKK.  [1 1] 

A  group  of  suspected  PKK  fighters  attacked  a  Turkish  Army  convoy 
with  rocket-propelled  grenades  and  small  arms  fire  in  the  country’s 
southeast.  At  least  one  bus  was  completely  destroyed,  killing  10 
soldiers  and  leaving  more  than  60  others  injured.  Witnesses  later 
reported  seeing  military  F-16  jets  taking  off  from  the  air  base  in 
Diyarbakir.  [12] 

Detection  of  terrorist  threats  is  essential  for  the  safety  of  borders,  assets 
along  the  borders,  and  security  of  the  whole  country.  The  goal  is  to  completely 
prevent  terrorist  groups  from  accessing  Turkey  through  the  border.  Once  this 
step  is  taken,  the  majority  of  the  terrorists’  resources  will  be  confined. 

B.  RESEARCH  QUESTIONS 

This  research  is  guided  by  the  following  questions: 

1.  Can  Unmanned  Aerial  Vehicles  (UAVs)  help  detect  and  classify 
terrorists  over  a  region  characterized  by  rough  topographical 
conditions  and  active  terrorists? 

2.  Can  UAVs  provide  early  warnings  of  terrorist  attacks,  and  how  many 
terrorists  can  be  detected? 

3.  What  number  of  UAVs  and  performance  characteristics  best  contribute 
to  border  security? 
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4.  How  do  changes  in  our  scenario  affect  the  ability  of  the  UAVs  to 
provide  early  warning? 

C.  SCOPE  OF  THE  THESIS 

UAVs,  aerial  vehicles  that  can  be  piloted  remotely,  are  a  critical 
component  of  modern  day  reconnaissance  and  surveillance.  UAVs  came  onto 
the  stage  in  the  1990s.  UAVs  successfully  addressed  the  deficiency  of  manned 
ISR  assets  in  the  Gulf  War  by  providing  enhanced  intelligence,  surveillance,  and 
reconnaissance  (ISR)  capabilities  [13],  [14].  Additionally,  the  DoD  Report  on 
Desert  STORM  points  out  that  “UAVs  proved  excellent  at  providing  an 
immediately  responsive  intelligence  collection  capability”  [13],  [15].  Technical 
capabilities  of  UAVs  can  be  used  to  improve  coverage  along  borders.  Advanced 
Electro-Optical  (E-O)  identification  technology,  UAV  loiter  capability,  the  range  of 
UAVs  compared  to  other  patrol  assets,  and  their  low  price  explains  why  UAVs 
should  play  a  key  role  in  ISR  missions.  The  growing  demand  for  UAVs  is  also 
based  on  avoiding  military  casualties.  In  terms  of  today’s  casualty  averse  military 
environment,  the  consequences  of  losing  an  airman  and  a  UAV  cannot  be 
compared.  Manned  systems  and  UAVs  can  accomplish  many  of  the  same  tasks. 
But  “UAVs  have  gained  favor  as  a  way  to  reduce  risk  to  combat  troops,  the  cost 
of  hardware,  and  the  reaction  time  in  a  surgical  strike”  [16],  [17]  and  “to  conduct 
missions  in  areas  that  are  difficult  to  access  or  otherwise  considered  too  high-risk 
for  manned  aircraft  or  personnel  on  the  ground”  [16],  [18]. 

UAVs  can  be  designed  to  fly  under  extreme  conditions  that  are  risky  for 
pilots.  They  can  fly  over  extreme  altitudes  for  extended  periods  without  suffering 
the  emotional  and  physical  effects  experienced  by  humans.  UAVs  also  improve 
situational  awareness.  Hence,  the  reason  unmanned  air  systems  (UAS)  appeal 
to  the  military  is  explained  in  Department  of  Defense  Appropriations  Bill  of  Fiscal 
Year  2003  as  follows: 

In  today’s  military,  unmanned  systems  are  highly  desired  by 
combatant  commanders  for  their  versatility  and  persistence.  By 
performing  tasks  such  as  surveillance;  signals  intelligence 
(SIGINT);  precision  target  designation;  mine  detection;  and 
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chemical,  biological,  radiological,  nuclear  (CBRN)  reconnaissance, 
unmanned  systems  have  made  key  contributions  to  the  Global  War 
on  Terror.  [19] 

However,  their  effectiveness  is  highly  dependent  on  the  characteristics  of 
the  region  in  which  they  are  deployed.  This  thesis  examines  the  effectiveness  of 
UAVs  in  helping  secure  a  border  characterized  by  rough  terrain  and  active 
terrorists.  For  this  purpose,  87  kilometers  of  Turkey’s  border  with  Iraq  is  modeled 
in  MANA  (Map  Aware  Non-Uniform  Automata).  The  impact  of  UAVs  on  detection 
and  classification  of  terrorists  who  use  the  southeast  border,  characterized  by 
rough  terrain,  as  passage  from  Northern  Iraq  into  Turkey  is  examined  by  varying 
the  following  factors  in  the  model:  the  number  of  UAVs,  UAV  sensor  range,  UAV 
speed,  number  of  terrorists,  detection,  classification,  and  communication  range  of 
terrorists  and  scouts,  etcetera.  This  analysis  explains  the  relationships  between  a 
variety  of  input  factors  used  in  the  detection  and  classification  process  and 
performance  measures. 

The  rough  topography  of  the  region  can  be  seen  in  Figure  1  and  Figure  2, 
which  depict  sections  of  the  southeast  part  of  Turkey.  Terrorists  have  to  walk 
long  distances  in  rough  terrain  while  crossing  the  border.  Due  to  these 
challenging  conditions,  terrorists  need  plenty  of  time  to  reach  their  destination. 
This  time  period  can  be  turned  into  an  advantage  with  an  efficient  UAV  network 
over  the  region  to  enhance  situational  awareness  and  real-time  imagery.  UAVs 
can  detect  terrorists  before  they  attack  Turkish  assets.  They  can  even  interrupt 
the  logistic  and  personnel  support  chain  of  terrorists.  The  priceless  benefit  of 
using  UAVs  can  be  a  significant  decrease  in  losses  from  terrorist  attacks. 
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Figure  1 .  Topographic  Map  of  Northern  Iraq  and  SE  Turkey.  Image  from 

positivity.wordpress.com. 
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Figure  2.  A  picture  from  the  SE  border.  Image  from  Milliyet.com.tr. 

D.  LITERATURE  REVIEW 

A  literature  review  on  border  security,  terrorist  activities,  and  UAVs  reveals 
that  many  students  at  NPS  have  studied  similar  problems  using  the  simulation 
environment  Map  Aware  Non  Uniform  Automata  (MANA).  Seven  studies,  four 
NPS  student  theses  and  three  other  research  studies  that  address  similar 
problems,  are  reviewed  below. 

In  his  thesis,  Oh  (2010)  examines  the  border  security  problem  of  the  ROK 
Army  [20],  He  develops  a  model  in  MANA  to  measure  the  time  needed  by  an 
enemy  to  reach  a  waypoint  and  the  probability  of  enemy  mission  failure.  The 
results  of  his  study  provide  insights  on  deciding  a  structure  for  border  security. 

The  Department  of  the  Prime  Minister  and  Cabinet  of  New  Zealand 
published  the  Maritime  Patrol  Review  (MPR)  in  2001.  The  review,  which  was 
driven  by  the  planned  $600M  sensor  system  upgrade  to  the  RNZAF’s  P-3  Orion 
maritime  patrol  aircraft,  highlighted  the  poor  state  of  maritime  domain  awareness 
in  New  Zealand  in  general  and  of  maritime  aerial  surveillance  in  particular.  Illegal 
fishing,  drug  smuggling,  illegal  immigration,  terrorist  activity,  energy  security,  and 
transnational  crime  are  some  of  the  many  threats  to  their  maritime  security.  In  his 
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thesis,  Oliver  (2009)  examines  the  effects  of  UAVs  on  New  Zealand’s  maritime 
security  and  claims  that  UAVs  provide  a  credible  option  to  manned  aircraft  [21]. 

In  his  thesis,  Yildiz  (2009)  explores  the  use  of  mini  Unmanned  Aerial 
Vehicles  (mini  UAVs)  along  with  other  assets  to  enhance  border  protection  [22], 
A  part  of  the  Tucson  sector  in  Arizona  is  modeled  in  an  agent-based  model 
(MANA)  for  this  purpose.  The  result  of  his  study  shows  that  the  use  of  mini  UAVs 
is  useful  in  interdiction  of  illegal  entrants  and  provides  an  enhancement  of  border 
security. 

In  his  thesis,  Sulewski  (2005)  analyzes  the  effect  of  UAVs  in  the  Army’s 
Future  Combat  Systems  family  of  systems  [23],  He  builds  a  simulation  model  to 
determine  how  the  numbers  and  the  capabilities  of  UAVs  affect  a  Future  Force 
Combined  Arms  Battalion’s  (CAB’s)  ability  to  secure  an  objective.  He  conducts 
46,440  computational  experiments  by  varying  many  factors,  including  UAV 
capabilities.  He  finds  that  the  UAVs,  their  capabilities,  and  tactics  are  significant 
factors  on  a  CAB’s  performance. 

In  his  study,  Raffetto  (2004)  develops  a  model  in  MANA  to  analyze  the 
effect  of  UAV  characteristics  on  ISR  missions  for  a  Marine  Expeditionary  Brigade 
(MEB)  commander  in  2015  [24],  As  a  result,  he  contends  that  the  UAV 
characteristics,  such  as  airspeed,  endurance,  sweep  width,  and  sensor  capability 
are  beneficial  in  terms  of  intelligence  gathering. 

Bertsche  and  Schwarz  (2002)  develop  a  model  in  MANA  to  analyze 
another  scenario:  “Protection  of  Stationary  Potential  Terrorist  Targets”  [25],  They 
state  that  agent-based  models  offer  a  better  solution  for  modeling  the  problem 
where  traditional  operation  research  (OR)  analysis  is  limited  in  explaining 
intangible  factors  such  as  persistence  and  courage.  The  results  of  their  study 
yield  significant  results  for  the  scenarios  studied. 

Butler’s  (2001)  research  report  indicates  that  UAVs  with  advanced 
sensors  are  an  integral  part  of  Twenty-first  Century  ISR  missions  [13].  The 
results  of  the  study  show  that  the  U.S  Air  Force  should  continue  to  use  UAVs  for 
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ISR  missions  and  improve  their  sensor  capabilities  in  accordance  with  rapid 
technological  development. 
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II.  MANA 


This  chapter  discusses  the  agent-based  modeling  platform  Map  Aware 
Non  Uniform  Automata  (MANA),  which  is  used  for  this  study.  First,  we  provide 
the  definition  and  benefits  of  agent-based  models.  Second,  we  present  the 
definition  and  evolution  of  advances  in  MANA,  with  several  example  studies. 
Finally,  we  explain  why  MANA  is  chosen  as  the  modeling  platform  to  conduct  this 
study. 

A.  AGENT-BASED  MODELING 

Agent-based  modeling  and  simulation  (ABMS)  has  gained  prominence 
lately  with  the  growing  interest  in  agent-based  modeling,  sophisticated  modeling 
software,  advanced  computer  technology,  and  the  need  for  data.  ABMS  is  used 
to  model  complex  systems  consisting  of  autonomous  agents  that  interact  with 
other  agents  and  their  environment.  Sanchez  and  Thomas  describe  agent-based 
simulations  as  models  in  which  multiple  entities  sense  and  stochastically  respond 
to  other  agents  and  the  conditions  in  their  local  environments,  mimicking  complex 
large  scale  system  behavior  [26],  In  an  agent-based  model,  autonomous 
decision  making  entities  (which  can  be  people,  vehicles,  ships,  aircraft,  animals, 
etc.)  follow  a  set  of  predefined  rules  and  algorithms  that  are  coded  based  on 
personality  weightings,  movement  algorithms,  and  penalty  functions.  Global 
behaviors  then  arise  as  a  result  of  interactions  of  individual  relationships. 

The  benefits  of  ABMS,  over  other  modeling  techniques,  can  be  explained 
in  the  following  three  statements  [27]: 

1 .  ABMS  captures  emergent  behaviors  resulting  from  the  interactions  of 
individual  entities. 

2.  ABMS  is  the  most  natural  way  of  describing  and  simulating  a  system 
composed  of  “behavioral”  entities.  Agent  behaviors  can  be  represented 
explicitly  within  a  range  of  options. 

3.  ABMS  provides  flexibility  along  multiple  dimensions.  One  can  use 
ABMS  when  the  complexity  is  unknown  and  requires  some  effort. 
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Decision  makers  need  detailed  information  on  today’s  high  technology 
products,  many  of  which  are  costly  to  acquire.  Modeling  and  simulation  play  a 
key  role  in  gaining  insights  about  these  systems  under  limited  budget  and  time 
when  actual  data  is  not  available  or  limited.  MANA-V  (Map  Aware  Non-Uniform 
Automata-Vector),  an  agent-based  distillation  model  [28]  that  is  broadly  used  in 
military  operations  analysis,  is  used  to  model  the  border  area  in  this  study. 
Figure  3  provides  the  Terms-of-Use  screen  and  general  information  about  the 
developers  and  the  version  of  MANA.  The  following  sections  provide  an  overview 
of  MANA  based  mostly  on  the  user’s  manual. 


MANA  V 


DTA** 


DEFENCE  TECHNOLOGY  AGENCY 


Vector-Based  Movement 


Map  Aware  Non-uniform  Automata 

This  software  has  been  developed  by  Gregory  McIntosh.  David  Galliqan. 

Mark  Anderson  and  Michael  Lauren  for  the  New  Zealand  Army  and  Defence  Force. 

Terms  of  use: 

This  software  belongs  to  the  Defence  Technology  Agency  (DTA),  New  Zealand 
Defence  Force.  It  is  not  to  be  distributed  without  the  permission  of  DTA.  This 
product  is  not  to  be  offered  for  resale. 


View  end  user  license  agreement  (EULM 


See  https://teams. nzdf.mil. nzisites/mana  for  updates  and  support. 

(Requires  user  account.) 


I  understand  I  can  only  use  this  software  subject  to  the  conditions  stated  above. 


i/ersion  5.01.03 


I  Agree 


Cancel 


pic:  J  Peetc 


Figure  3.  MANA  terms-of-use  screen. 
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B.  EVOLUTION  OF  MANA 


MANA  is  a  stochastic,  time-step  software  package  designed  to  facilitate 
building  and  conducting  combat  simulations.  MANA  has  been  and  continues  to 
be  developed  by  the  Operations  Analysis  Section  of  New  Zealand’s  Defense 
Technology  Agency  (DTA).  DTA  was  inspired  by  the  pioneering  efforts  of 
llachinski’s  agent-based  models  ISAAC  and  EINSTein  [28],  These  models 
helped  DTA  to  see  that  a  small  scale,  flexible  model  can  be  more  suitable  than  a 
detailed  simulation  for  the  requirements  of  the  New  Zealand  Defense  Force 
(NZDF).  Additionally,  DTA  realizes  that  automaton  models  can  be  used  for 
analysis  purposes.  Nonetheless,  DTA  is  well  aware  that  they  need  to  improve 
automaton  style  models  to  fully  replace  the  older  large  scale  models. 

In  2000,  DTA  took  the  first  step  of  improvement  with  the  “Situational 
Awareness  Map,”  using  MATLAB.  Afterwards,  the  programming  language 
changed  with  the  more  flexible  programming  language  Delphi.  In  general,  MANA 
is  designed  as  a  scenario  analysis  model.  Situational  awareness,  communication 
links,  terrain  map,  waypoints,  and  event-driven  personality  changes  are  the 
strengths  of  MANA  over  most  other  agent-based  models.  The  evolutionary  steps 
of  MANA  are  as  follows  [29], 

1.  MANA  2  (2002/2003) 

As  the  first  available  version  of  MANA,  this  version  forms  the  basis  of 
ensuing  models.  MANA  2  consists  of  primary  movement  weightings,  terrain 
editing  features,  sensor  and  weapon  characteristics:  simple  cookie-cutter 
scheme  or  with  tables  of  range-dependent  probabilities,  squad-based  agent 
property  definition,  and  shared  situational  awareness  (SA)  maps.  SA  maps  can 
be  used  to  model  fundamental  aspects  of  Network  Centric  Warfare  (NCW). 

Wolf  (2003)  used  MANA  version  2  to  build  his  model  about  logistic  support 
operations  [30],  He  analyzed  the  potential  of  ABMs  for  logistical  decision  making 
and  mission  success.  He  claims  that  ABMs  are  very  useful  in  exploring  highly 
complex  scenarios. 
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2.  MANA  3  (2003/2004) 

A  detailed  NCW  is  possible  in  this  version  with  an  improvement  to  the 
squad  SA  map,  communication  links,  and  information  sharing  between  agents. 
Supplemental  movement  algorithms  allow  agents  to  determine  their  movement 
based  on  information  flow  between  other  agents.  Expanded  data  farming 
capabilities,  special  aircraft  algorithms,  and  search  algorithms  are  also 
introduced  with  this  version. 

Pfeiffer  (2006)  used  MANA  version  3.2.1  to  exam  the  various  factors  that 
affect  convoy  missions  in  an  urban  environment  [31].  He  claims  that  MANA  can 
produce  useful  insight  even  though  it  has  limitations.  Tiburcio  (2005)  utilizes 
MANA  version  3.0.37  in  his  study  of  maritime  protection  of  critical  infrastructure 
assets  in  the  Campeche  Sound  [32],  Cason  (2004)  developed  a  simulation  model 
using  MANA  version  3  to  reveal  significant  factors  of  UAVs  in  an  urban  infantry 
patrolling  operation  [33],  Aydin  (2004)  also  conducts  his  study,  “An  Exploration 
Analysis  of  Village  Search  Operations,”  using  MANA  version  3.0.  He  investigates 
a  village  search  operation  that  takes  place  in  southeastern  Turkey.  He 
commends  the  efficiency  of  MANA  “at  exploring  the  factors  affecting  the  non¬ 
linear  nature  of  a  search  operation  and  the  emergent  behavior  in  low  intensity 
conflicts”  [34], 

3.  MANA  4  (2005/2006) 

“Human-in-the-loop”  studies  are  now  possible  with  the  new  data  streaming 
capability  of  this  version.  A  battlefield  zoom  property,  a  genetic  algorithm,  a  data 
analysis  tool,  finite  sensor  and  weapon  apertures,  angular  additions  to  the 
movement  algorithms  that  include  direction  of  facing,  and  squad  formation 
shapes  are  the  primary  additional  features  of  MANA  version  4. 

Oh  (2010)  utilizes  MANA  version  4  in  his  study  on  ROK  Army  border 
security  problems  because  of  the  limitations  in  the  alternatives  [20],  Singham, 
Therkildsen,  and  Schruben  use  MANA  version  4  in  their  analysis  on  flocking 
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algorithms  to  input  modeling  for  agent  movement  [35],  Yildiz  (2009)  develops  a 
model  using  MANA  version  4.04.1  for  his  study  [22], 

4.  MANA  5  (2006/2013) 

There  is  a  conceptual  change  in  this  latest  version.  Grid-based  movement 
algorithms,  which  are  used  in  all  previous  versions  of  MANA,  are  now  converted 
to  vector  based  movement  algorithms.  Figure  3  provides  insight  into  both 
approaches.  In  previous  versions  of  MANA,  agents  in  the  battlefield  move  grid 
square  to  grid  square  in  each  time  step  depending  on  model  set-up.  The  next 
grid  that  an  agent  is  about  to  move  can  involve  enemy,  neutral,  friendly  units,  and 
terrain  features.  The  user  defined  personalities  constitute  the  basis  of  agent 
behaviors. 

In  the  latest  version,  agents  in  the  battlefield  calculate  a  vector,  such  as 
Fe,  Ft,  Fw  given  in  Figure  4,  toward  all  other  entities  of  interest  and  terrain  within 
sensor  range  or  depending  on  information  provided  by  inorganic  contacts.  The 
product  of  these  vectors  and  the  personality  weightings  generate  the  final  vector. 
The  agent  movement  is  calculated  based  on  Newton’s  second  law  and  standard 
kinematic  equations  for  constant  acceleration. 


Figure  4.  Two  different  approaches  to  calculate  movement:  The  grid-based 
(right),  the  vector  based  (left)  (From:  MANA  user  manual). 
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The  advantages  of  vector  based  algorithms  are  explained  in  the  MANA 
user  manual  as  follows: 

1 .  Larger  battlefields  can  be  modeled. 

2.  Battlefield  region,  agents  speeds,  sensor,  and  weapon  characteristics 
can  be  defined  in  terms  of  real  world  units.  Therefore,  MANA  users  no 
longer  have  to  convert  real-world  units  to  model  units. 

3.  New  features  can  be  easily  added  to  models  and  scenarios. 

The  rest  of  the  model  properties  remain  almost  unchanged.  Therefore,  the 
developers  publish  the  MANA-V  user  manual  as  a  supplementary  document  to 
the  MANA  4  user  manual. 

C.  WHY  MANA 

MANA  version  5.01.03  is  chosen  as  the  modeling  tool  to  support  this 
research.  In  MANA,  agents  are  aware  of  their  local  environment,  terrain,  and 
battlefield  activities.  Agent  behaviors  are  determined  based  on  their  activities, 
goals,  and  terrain  type.  However,  all  agents  are  not  necessarily  reacting  in  the 
same  manner.  They  respond  to  events  according  to  personality  weightings  and 
information  provided  by  organic  and/or  inorganic  SA  maps. 

MANA  requires  a  shorter  training  period  than  most  other  comparable 
modeling  environments.  The  simple  graphical  user  interface  (GUI)  allows  users 
to  build  scenarios  in  a  short  time  period.  Furthermore,  MANA  has  an  “on-the-fly” 
editing  capability.  Users  can  edit  scenarios  at  the  time  updates  are  needed. 
Primarily,  analysts  prefer  MANA  because  of  their  limited  time  to  grasp  dynamics 
of  the  real  life  situations  for  programming  into  higher  resolution  models. 

Agent  behaviors  in  MANA  are  controlled  by  decision  making  algorithms. 
Therefore,  agents  react  based  on  personalities,  not  orders.  The  interaction  of 
agents  can  be  analyzed  in  MANA.  Agents  may  change  their  personalities  as  a 
result  of  triggering  events.  Built-in  data  farming  capabilities  allow  users  to  readily 
explore  parameter  variables  in  their  model. 
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Unlike  other  highly  detailed  agent-based  models,  MANA  reflects  only  the 
essential  details  of  a  scenario.  “It  is  a  myth  that  a  more  detailed  model  is 
necessarily  a  better  model,  because  it  is  impossible  to  capture  accurately  every 
aspect  of  nature”  [28],  The  more  complex  a  model  is  the  less  transparent  it  is. 
Hence,  it  would  be  difficult  to  model  local  relationships  of  this  geography  based 
analysis  and  to  reveal  interactions  between  factors  using  a  conventional  model. 
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III.  SCENARIO  AND  MODEL  DEVOLOPMENT 


“All  models  are  wrong,  but  some  are  useful.  ” 

-George  Box 

In  this  chapter,  we  provide  detailed  information  on  scenario  and  model 
development.  First,  we  explain  a  real-life  skirmish  that  motivates  the  baseline 
scenario  analyzed  in  this  study.  Second,  we  cover  all  models  developed  based 
on  this  skirmish.  Third,  we  describe  the  battlefield  and  terrain  features.  Finally, 
we  introduce  the  agents  in  the  model. 

A.  SCENARIO 

The  baseline  scenario  analyzed  in  this  research  is  based  on  a  skirmish 
between  a  guerilla  team  and  a  Turkish  battalion  in  the  Southeast  region  of 
Turkey.  In  October  2007,  an  estimated  150  to  200  terrorists  attacked  an  infantry 
battalion  located  near  the  Southeast  border  of  Turkey  with  Iraq.  The  nearest 
friendly  force  was  situated  5.5km  directly  behind  the  attacked  battalion.  Because 
of  the  hard  terrain,  the  two  battalions  were  connected  to  each  other  via  a  bridge, 
which  was  the  only  way  of  land  transport,  over  a  river. 

Using  information  from  news  reports  of  the  attack,  the  Blue  forces,  which 

represent  the  friendly  side  in  this  scenario,  are  composed  of  50  infantrymen,  two 

cannons,  rocket  projectors,  and  grenades.  The  commander  positions  the 

personnel  to  watch  the  area  for  terrorist  activity.  Figure  5  depicts  a  picture  of  the 

attacked  battalion.  Field-of-vision  is  a  major  problem  when  considering  the 

rugged  characteristic  of  the  province,  especially  during  the  night.  The  Red  forces, 

which  represent  the  terrorists,  are  composed  of  150  to  200  guerillas  equipped 

with  grenades,  bazookas,  and  other  small  arms.  The  terrorists  split  into  three 

groups.  Two  of  the  groups  carry  out  missions  before  commencement  of  the  main 

assault.  The  purpose  of  the  remaining  terrorists  is  to  directly  attack  the  battalion. 

The  Blue  forces  are  well  trained  and  equipped,  but  the  Red  side  has  the 

advantage  of  easy  adaptation  to  rough  terrain.  The  terrorists  use  the  terrain  to 
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increase  their  effectiveness.  Additionally,  the  Blue  side  has  no  intelligence  about 
the  attack,  so  the  terrorists  have  the  advantage  of  surprise. 


Figure  5.  A  view  of  the  attacked  battalion  (From:  Haberler.com). 

The  Red  side  conducts  reconnaissance  activities  to  gather  necessary 
information  and  intelligence  about  the  region  and  Blue  forces,  and  plan  their 
attack  based  on  this  information.  The  length  of  the  pre-attack  period  mostly 
depends  on  the  infiltrators,  who  stay  covered  all  the  time  and  act  as  neutral 
parties.  Since  the  Red  commander  wants  to  ensure  the  success  of  the  attack,  he 
assigns  three  of  his  subordinates  as  a  leader  for  each  group.  Following  the 
planning  step,  the  guerilla  team  enters  Turkey  through  the  northern  Iraq  border 
around  2330.  They  split  into  groups  as  planned,  and  the  first  group  of  10  guerillas 
goes  to  the  village  and  cuts  the  electricity  and  telephone  lines  before 
commencement  of  the  attack.  The  second  group  of  20  guerillas  blows  up  the  only 
bridge  to  the  outpost.  Then,  these  two  groups  combine  into  a  single  team  to  stop 
or  delay  any  reinforcement.  Meanwhile,  the  third  group  proceeds  to  their  initial 
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position.  Once  they  take  position,  four  of  the  terrorists  form  a  recon  team  and 
start  close  defensive  fire  at  around  0200  to  reveal  the  location  of  the  battalion  so 
that  the  attackers  located  on  another  mountain  across  from  the  battalion  can  aim 
their  shots.  One  of  the  critical  issues  for  the  Red  side  is  time.  In  general,  guerilla 
teams  avoid  long  engagements.  But,  in  this  case,  they  are  well-prepared  and 
outnumber  the  isolated  Blue  side  by  a  factor  of  four.  So,  the  plan  is  to  overrun  the 
outpost  before  reinforcements  arrive.  Conversely,  the  Blue  side  tries  to  stay 
close,  with  a  minimum  of  friendly  losses,  to  keep  coordination  and 
communication  as  effective  as  possible,  and  to  hinder  the  Red  side  from 
capturing  the  battalion. 

The  salient  point  of  the  summary  is  the  Blue  force’s  lack  of  intelligence 
and  detection  until  Red  attacks.  This  study  of  the  baseline  scenario  focuses  on 
detection  of  Red  members  before  they  cross  the  border  or  before  they  reach  their 
attack  points.  Therefore,  no  skirmish  between  the  Red  and  Blue  sides  is  explicitly 
modeled.  For  this  purpose,  five  different  versions  of  the  scenario  are  analyzed  to 
derive  the  effectiveness  of  Unmanned  Aerial  Vehicles  (UAVs)  (from  0  up  to  4) 
assigned  over  the  area  of  interest. 

1.  Baseline  Scenario 

This  scenario  is  based  on  the  skirmish  explained  above.  This  baseline 
scenario  includes  all  of  the  agents  except  UAVs.  There  are  150  terrorists,  divided 
in  four  groups,  which  try  to  cross  the  border  and  make  their  way  to  their  attack 
positions.  Initial  runs  of  the  baseline  scenario  help  us  quantify  how  many  of  the 
Red  Agents  will  reach  the  final  waypoint  without  UAV  detection,  as  in  reality, 
during  the  predefined  maximum  time  limit.  Even  though  Scouts,  Civilians,  and  a 
Blue  Killing  Agent  exist  in  the  scenario,  they  do  not  have  any  effect  on  the  result 
for  this  specific  scenario. 


21 


2. 


Scenario  Two 


There  is  only  one  area  of  responsibility  (AOR)  and  one  UAV  in  this 
scenario.  This  is  the  extension  of  the  baseline  scenario  with  one  UAV  over  the 
whole  area  of  interest.  The  screenshot  in  Figure  6  depicts  the  placement  of  one 
UAV  and  the  other  agents.  The  terrain  types  are  modeled  using  different  colors. 
The  roads  are  colored  as  yellow.  The  dark  brown  and  light  brown  colors 
represent  the  mountains  and  off-road  areas.  We  add  a  grey  colored  layer  to  the 
terrain  to  facilitate  visualization  of  the  border  region. 
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Figure  6.  Scenario  two  developed  using  MANA. 


3.  Scenario  Three 

The  baseline  scenario  was  again  updated,  this  time  adding  two  AORs  and 
two  UAVs  over  the  region.  As  in  the  second  extension  of  the  baseline  scenario, 
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the  third  scenario  allows  us  to  see  how  the  number  of  Blue  and  Red  casualties 
changes  depending  on  the  number  of  UAVs,  two  in  this  case.  The  screenshot  in 
Figure  7  shows  the  position  of  the  two  UAVs,  based  on  the  AORs,  and  the  other 
agents. 
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Figure  7.  Scenario  three  developed  using  MANA. 


4.  Scenario  Four 

Scenario  four,  the  fourth  updated  version  of  the  baseline  scenario, 
consists  of  three  AORs  and  three  UAVs.  There  are  no  other  differences  between 
the  scenarios  in  terms  of  agent  placement  or  parameter  and  factor  set  up.  This 
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scenario  allows  us  to  see  the  effect  of  three  UAVs,  shown  in  Figure  8,  on  the 
results. 


#  MANA-Scenario.Four.3AOR 


Figure  8.  Scenario  four  developed  using  MANA. 

5.  Scenario  Five 

As  the  final  extended  version  of  the  baseline  scenario,  the  fifth  scenario, 
comprises  four  AORs  and  four  UAVs.  The  only  difference  between  the  modified 
scenarios  and  baseline  scenario  is  the  number  of  AORs  and  the  number  of 
UAVs.  The  purpose  of  Scenarios  Two  though  Five  is  to  see  how  the  outcome  of 
the  model  changes  with  the  number  of  UAVs.  Therefore,  all  other  aspects  of  the 
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model,  including  the  design  of  experiment,  are  kept  constant.  The  screen  capture 
in  Figure  9  displays  the  positions  of  UAVs  on  the  battlefield  for  this  scenario. 


ft  MANA  -  Scenario_Five_4AOR 


Figure  9.  Scenario  five  developed  using  MANA. 

B.  BATTLEFIELD 

In  MANA,  the  battlefield  is  a  bounded  area  on  which  all  entities  are  placed. 
For  this  study,  the  battlefield  represents  an  87  x  108  km  border  region  in  the 
Southeast  part  of  Turkey.  Battlefield  distances,  sensor  ranges,  and  weapon 
ranges  are  defined  continuously  in  terms  of  real  world  units  (meters). 

The  battlefield  consists  of  three  types  of  maps:  Background,  Terrain,  and 
Elevation  maps.  The  background  map  is  used  on  top  of  the  battlefield  for 
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cosmetic  purposes  to  improve  visualization  of  the  scenario.  The  background  map 
has  no  effect  on  going,  concealment,  and  cover.  The  terrain  map  includes  terrain 
features  for  each  terrain  type  used  in  the  model,  such  as  roads,  mountains,  etc., 
that  agents  can  follow  and  use  for  concealment  or  cover.  Three  different  RGB 
color  codes  are  used  to  define  each  terrain  type.  Terrain  types  have  distinctive 
characteristics  in  terms  of  going,  concealment,  and  cover.  Possible  going, 
concealment,  and  cover  values  range  from  0.0  to  1.0  and  affect  agents’  speed, 
sensor  capability  for  detection  and  classification,  as  well  as  terrain  protection 
from  weapon  fires,  respectively.  Terrain  map  and  terrain  features  are  provided  in 
Figure  10. 
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Figure  10.  Terrain  map  and  terrain  features. 


The  elevation  map,  which  consists  of  greyscale  colors,  is  used  to  define 
elevation  features  of  the  terrain.  For  this  study,  the  real  world  elevation  range  is 
defined  from  640  meters  to  3185  meters.  The  elevation  map  used  in  this  study  is 
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depicted  in  Figure  1 1 .  The  white  and  black  colors  in  the  elevation  map  represent 
the  highest  and  lowest  point  of  the  region  respectively.  In  the  terrain  map,  the 
white  areas  are  formed  as  mountains.  Terrain  and  elevation  maps  affect  the 
speed  and  LOS  calculations  of  the  agents.  The  multipliers  given  for  the  terrain 
types  are  used  when  computing  the  movement  speed,  sensor  detection 
associated  with  the  concealment  factor,  and  hit  rate  that  affects  the  shot 
probabilities.  The  LOS  is  calculated  based  on  the  sensor  height  of  each  agent 
and  elevation  ranges  defined  for  the  elevation  map. 


Figure  1 1 .  Elevation  map. 


The  elevation  data  is  downloaded  from  the  EarthExplorer  website,  in 
Digital  Elevation  Model  (DEM)  format.  The  elevation  range  changes  between  640 
meters  to  3185  meters.  Then,  the  ArcMap,  a  part  of  the  ArcGIS  toolbox,  is  used 
to  view  the  elevation  data,  mark  off  the  desired  play  box,  make  note  of  the 
highest  and  lowest  points  of  elevation,  and  export  a  picture  of  it.  Finally, 
Paint. Net  is  used  to  convert  the  picture  to  bitmap  (BMP)  format  and  resize  it  so 
that  it  will  allow  adequate  resolution  for  the  selected  play  board. 
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The  ArcMap  and  EarthExplorer  are  again  used  to  acquire  different  views 
of  the  terrain  to  support  the  creation  of  the  terrain  map.  Paint. Net  and  Adobe 
Photoshop  CS6  software  are  used  to  generate  layers  for  different  terrain  types. 
The  final  product  is  flattened  into  a  BMP  image  that  MANA  understands. 

The  time  step  of  all  the  simulation  runs  is  set  at  five  seconds.  Initially,  the 
entire  model  was  developed  using  a  time  step  of  one  second.  A  time  step  of  one 
took  too  long  to  run.  Therefore,  experiments  were  conducted  with  the  values 
three,  five,  and  ten  to  see  how  the  model  runs  with  these  time  steps.  The  model 
runs  show  that  a  time  step  of  ten  seconds  is  an  ineffectual  candidate  because  it 
causes  the  model  to  change  states  in  an  unexplained  pattern.  So,  to  avoid 
incorrect  results,  a  time  step  of  five  seconds  is  selected.  However,  the  model  still 
requires  plenty  of  time  for  a  complete  run.  Consequently,  the  maximum  time  step 
is  defined  as  9000  time  steps.  This  corresponds  to  twelve  and  a  half  hours  in  real 
life.  The  baseline  model  yields  that  this  number  is  adequate  for  half  of  the  Red 
members,  on  average,  to  reach  their  final  waypoints. 

While  dealing  with  the  time  step,  it  was  found  that  time  in  “Fuel  Out”  state 
is  not  defined  in  terms  of  seconds  as  stated  in  the  “Edit  Squad  Properties”  panel 
and  user’s  manual.  In  fact,  the  duration  in  this  state  is  defined  as  time  steps. 
Additionally,  the  first  two  seconds  of  “Spare  1”  state,  which  is  used  for  refueling 
purposes  in  this  study,  is  ignored  and  refueling  starts  at  the  third  second.  (These 
problems  have  been  reported  to  the  developers.) 

C.  AGENT  DESCRIPTIONS 

Squads  constitute  the  key  unit  of  Map  Aware  Non  Uniform  Automata 
(MANA).  The  maximum  number  of  squads  in  a  scenario  is  32,767.  The  number 
of  agents  in  a  squad  can  be  between  1  and  1000.  There  are  13  squads  in  this 
study,  as  shown  in  Table  1 . 
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Table  1 .  Agents  and  features. 


Squad 

Name 

Allegiance 

Threat  level 

Class 

1 

Blue UAVl 

1 

3 

1 

2 

Blue UAV2 

1 

3 

1 

3 

Blue UAV3 

1 

3 

1 

4 

Blue UAV4 

1 

3 

1 

5 

Blue  Battalion 

1 

3 

0 

6 

Red Teaml 

2 

3 

2 

7 

Red Team2 

2 

3 

2 

8 

Red Team3 

2 

3 

2 

9 

Red Team4 

2 

3 

2 

10 

Red  Scouts 

2 

3 

3 

11 

Civilians  (Neutrals) 

0 

2 

4 

12 

Blue  Killing  Agent 

1 

3 

5 

13 

Blue  Targets 

1 

2 

6 

The  allegiance  represents  the  agent’s  side  in  the  battle.  The  allegiance  of 
an  agent  can  be  friend,  enemy,  or  neutral.  Threat  levels  and  class  parameters 
are  used  to  define  different  types  of  agents  in  a  scenario.  For  example,  UAVs 
and  Blue  Killing  Agent  are  allied,  but  they  are  equipped  differently,  and  their  task 
in  the  scenario  is  also  different.  So,  threat  levels  and  class  parameters  help 
distinguish  these  agents  in  the  scenario. 

The  placement  of  agents  onto  the  battle  field  is  shown  in  Figure  12.  The 
Red  Team,  divided  in  four  groups,  starts  on  the  Iraq  side  of  the  border.  Scouts 
are  located  along  the  border  so  that  they  can  provide  information  on  UAV 
activities.  The  Civilians  are  distributed  around  the  battlefield.  Each  UAV,  which  is 
responsible  for  predefined  area  of  responsibility,  is  defined  as  an  airborne 
sensor. 
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Figure  12.  Overview  of  battlefield  with  all  agents. 


1.  Blue  UAVs 

Blue  UAVs  are  responsible  for  monitoring  the  87  km  border,  which  is 
divided  into  AORs  (from  0  up  to  4).  The  personality  weightings  and  trigger  states 
for  UAVs  are  shown  in  Figure  13.  The  personality  weightings  are  set  using  a 
slide  bar  and  can  take  values  between  -100  and  100.  Positive  values  indicate  a 
positive  propensity  towards  the  associated  personality,  while  negative  values 
indicate  a  negative  propensity.  Personality  settings  other  than  the  default  value  of 
zero  are  shown  in  red.  To  ensure  that  UAVs  will  not  engage  the  same  group  of 
Red  Agent,  personality  weightings  towards  friends  are  set  to  -100  in  all  states.  At 
the  beginning  of  each  model  run,  UAVs  directly  fall  into  the  run  start  state,  which 
allows  UAVs  to  start  patrolling  at  random  and  different  times.  Each  UAV  is 
capable  of  detecting,  classifying,  and  tracking  illegal  entrants.  A  UAV  follows 
predefined  waypoints  in  the  default  state.  Once  it  detects  an  activity,  it  then 
proceeds  to  the  detection  area  for  classification.  The  UAV  tracks  the  classified 
agent  if  it  is  a  Red  Agent  and  sends  the  information  to  the  Blue  Killing  Agent. 
Otherwise,  the  UAV  keeps  following  its  waypoints. 
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Figure  13.  Blue  UAV  squad  personalities  and  trigger  states. 


The  Inorganic  Squad  SA  panel  controls  the  flow  of  information  among  the 
squads.  The  communication  link  parameters  are  invariant  within  the  trigger 
states.  UAVs  have  a  communication  capability  with  the  Blue  Killing  Agent  as 
shown  in  Figure  14.  To  prevent  multiple  detections,  the  communication  latency 
parameter  is  set  to  zero. 
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Figure  14.  Blue  UAV  inorganic  SA  communication  link  parameters. 


There  is  no  specific  refueling  strategy  for  the  UAVs.  When  low  on  fuel, 
they  jump  into  the  “Fuel  Out”  state  and  go  to  their  alternative  waypoint  for 
refueling.  UAVs  have  a  strong  propensity  to  go  directly  to  their  alternative 
waypoint  in  Fuel  Out  state  (Alternative  Waypoint  weighting  is  set  to  100).  The 
refueling  state  has  the  highest  priority.  Therefore,  UAVs  cannot  be  interrupted  by 
any  other  activity. 

2.  Blue  Battalions 

There  are  two  Blue  Battalions,  which  provide  final  waypoints  for  Red 
Agents.  The  final  waypoints  for  the  Red  Agents  are  set  depending  on  the  location 
of  the  battalions. 
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3 


Red  Teams 


Red  Teams  represent  the  terrorists.  There  are  four  groups  of  Red  Teams 
trying  to  cross  the  border.  The  personality  weightings  and  trigger  states  are 
shown  in  Figure  15.  Initially,  each  group  is  located  at  four  different  points  along 
the  Iraq  side  of  the  border.  The  Red  Teams  are  the  most  valuable  target,  with  a 
threat  level  of  three.  They  follow  their  waypoints  into  Turkey  and  try  to  avoid 
UAVs  along  their  path.  They  have  a  strong  negative  propensity  towards  UAVs 
(Enemy  Threat  3  weighting  is  set  to  -100)  and  a  desire  for  concealment. 


Figure  15.  Red  Agents  squad  personalities  and  trigger  states. 
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Red  Agents  have  no  communication  capability  with  other  agents,  but  they 
can  get  information  on  UAV  activities  via  Scouts  in  the  base  model  and  via 
Scouts  and  Civilians  in  the  additional  scenarios.  Once  Red  Agents  are  classified 
by  UAVs,  they  are  devolved  to  the  Blue  Killing  Agent. 

Red  Agents  increase  their  concealment  value  to  100%  in  all  enemy 
contact  states  except  for  the  Enemy  Contact  2  state  in  which  they  make  contact 
with  Blue  Targets.  The  experience  of  terrorist  teams  and  the  harsh  terrain  of  the 
region  allow  terrorists  to  have  a  perfect  stealth  value  in  most  cases.  This  higher 
concealment  value  allows  us  to  see  the  effectiveness  of  UAVs  in  worst  case 
conditions.  Once  they  reach  the  final  waypoint,  each  Red  Agent  kills  one  Blue 
Target  and  becomes  invisible  so  as  not  to  be  detected  and  killed.  This  allows  us 
to  count  how  many  terrorists  reach  their  destination  undetected  in  the  allotted 
time. 

Each  Red  Agent  has  a  weapon  that  can  fire  within  a  range  of  600  meters. 
The  Red  Agent’s  weapon  parameters  are  set  as  in  Figure  16.  The  amount  of 
ammunition  for  each  weapon  is  set  to  one  to  ensure  that  each  Red  Agent,  who 
reaches  the  final  waypoint,  will  kill  only  one  Blue  Target.  Civilians,  UAVs,  and 
Blue  Killing  Agent  are  selected  as  non-target  classes  to  prevent  the  Red  Agent 
from  firing  on  these  agents  and  consuming  their  ammunition.  In  this  model, 
attrition  is  used  only  to  record  how  many  Red  Agents  are  detected  by  a  UAV 
prior  to  reaching  their  destinations  and  how  many  make  it  to  their  destinations 
undetected. 
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Figure  16.  Red  Agents  weapon  settings. 

4.  Scouts 

There  are  17  Scouts,  who  are  invisible  to  all  other  entities  on  the 
battlefield,  distributed  along  the  border  to  provide  information  about  UAV 
activities  to  the  Red  Teams.  Scouts  have  a  stationary  observation  point.  They  are 
able  to  extend  their  sensor  range  by  using  binoculars.  The  inorganic  SA 
communication  links  for  Scouts  are  shown  in  Figure  17.  Scouts  can  communicate 
with  Red  Teams  to  provide  information  on  UAV  activities. 


35 


General  Map  Personalities  Tangibles  Sensors  Weapons  Intra  Sqd  SA  Inter  Sqd  SA  Advanced 

Inbound  Inoraanic  Information  A" times  arf in  battlefield  time  stePs- 

- 3 -  All  distances  are  in  metres. 


Outbound  Communication  Links  0  Master  Enable 


Squad 

Range  Capacity 

Buffer 

Latency 

Self 

Reliab. 

Acc. 

MxAge 

Rank  Filter 

Include 

Delivery 

□ 

8000 

50.0 

-1 

20 

5 

100.0 

100.0 

-1 

High 

ETC 

F-N-F 

7  8000  50.0  S)  |5  100.0  100.0  |-1  High  [ETC  |F-N-F 

8  8000  50.0  -1  20 

9  8000  \5QA  Li  So 


Create 

~1-Edit  JT 

L  Delete  | 

5  100.0  100.0  -1  High  ETC  F-N-F 

5  1 100.0  I00.0T1  High  OC  F-N-F 


BFuse  Unknowns  on  Inorganic  Map 
Fuse  Time:  0 
Fuse  Radius:  0 


© 

g  (metres) 


Lockout  Time  (All  Maps):  0 


© 


Min.  Link  Rank  Accepted:  Low 

Inorganic  Contact  Persistance:  6  ^ 

Force  Addition  of  New  Classified 
Contacts  to  Map 

Contact  Aggregation  Radius:  10 
(metres) 


Squad  #10  I  Jump  ]  Default  State 
Red  Scouts 


OK  Cancel 


MANAV 

Trigger  State 

Invariant 

Properties 


Figure  17.  Scouts  inorganic  SA  communication  link  parameters. 


5.  Neutrals 

Neutrals  are  distributed  around  the  battlefield  and  given  some  waypoints 
inside  the  AOR  of  UAVs.  Having  a  threat  level  of  two,  they  are  not  as  valuable  as 
Red  Agents.  Neutrals  are  the  agents  that  cause  distraction  to  the  UAVs  until  they 
are  classified  as  neutral  in  the  base  model.  However,  they  can  provide 
information  on  UAV  activities  to  the  Red  Agents  in  some  scenarios.  In  these 
scenarios,  Neutrals  communicate  with  Red  Agents  over  a  radio  and  take  the 
roles  of  distractor  and  infiltrator. 
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6.  Blue  Killing  Agent 

There  is  one  Blue  Killing  Agent  that  takes  classified  Red  Agents  out  of 
action.  The  Blue  Killing  Agent  has  a  perfect  concealment  value  and  remains 
invisible  during  the  whole  scenario.  In  reality,  the  information  on  terrorist 
activities,  provided  by  UAVs,  will  be  transferred  to  another  security  unit.  Current 
border  security  architecture  does  not  include  UAVs.  All  information  going  to  the 
Blue  Killing  Agent  flows  through  the  UAVs.  Therefore,  this  agent  is  used  to 
disregard  classified  Red  Agents  to  avoid  counting  multiple  detections  of  the  same 
Red  Agent.  The  Killing  Agent  has  a  perfectly  accurate  weapon  so  that  it  can  kill  a 
classified  Red  Agent  with  one  shot.  The  weapon  parameters  are  shown  in  Figure 
18.  The  weapon  fire  range  is  set  to  100,000  meters.  Red  Agents  and  Scouts  are 
selected  as  target  classes.  The  Blue  Killing  Agent  is  a  modeling  tool  that  kills  Red 
Agents  depending  on  the  information  provided  by  UAVs. 
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Figure  18.  Blue  Killing  Agent  weapon  settings. 


7.  Blue  Targets 

There  are  four  groups  of  Blue  Targets,  each  with  the  same  number  of 
agents  as  the  Red  Teams.  The  Blue  Targets  are  located  at  the  final  waypoints  of 
the  Red  Teams.  The  icon  type  for  this  squad  is  set  as  zero.  Thus,  they  cannot  be 
seen  by  the  users,  but  they  are  visible  to  all  battlefield  agents.  Having  a  threat 
level  of  two,  they  are  not  a  real  threat  for  the  Red  Agents.  They  do  not  have  any 
weapons  and  do  not  resist  Red  Agents.  Each  Blue  Target  is  killed  by  one  Red 
Agent,  who  has  only  one  bullet.  The  number  of  killed  Blue  Targets  forms  a 
surrogate  for  the  number  of  Red  Agents  who  can  reach  their  final  waypoint. 
There  are  150  terrorists  in  total,  and  there  will  be  undetected  terrorists  who 
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cannot  reach  the  final  waypoint  at  the  end  of  predefined  maximum  time  limit. 
Hence,  this  squad  is  required  to  count  the  number  of  terrorists  who  can  reach  the 
final  waypoint. 


39 


THIS  PAGE  IS  INTENTIONALLY  LEFT  BLANK 


40 


IV.  DESIGN  OF  EXPERIMENTS  AND  MODEL  RUNS 


Design  of  experiment  (DOE)  techniques  allow  for  obtaining  individual  and 
interactive  effects  of  factors  that  affect  the  results.  DOE  enables  a  robust  design 
by  providing  full  insight  into  interactions  between  design  factors.  Therefore,  DOE 
plays  a  key  role  in  data  collection  through  simulation  runs.  In  this  chapter,  we 
discuss  the  factors  and  levels  which  form  the  experimental  design  of  our  study. 
Additionally,  we  explain  the  tools  and  techniques  used  to  automatically  run  the 
model  given  the  experimental  design  on  the  Simulation  Experiments  and  Efficient 
Design  (SEED)  Center’s  (http://harvest.nps.edu)  computing  cluster.  Finally,  we 
provide  an  analysis  of  model  run  time  given  our  design. 

A.  FACTORS  AND  LEVELS 

A  wide  range  of  factors  can  be  varied  by  the  design  to  examine  their 
effects  on  the  outcome.  The  factor  ranges  were  selected  to  cover  the  capabilities 
of  existing  UAVs.  However,  it  is  practically  impossible  to  examine  every  possible 
factor  and  all  of  their  combinations.  Therefore,  we  conduct  our  analysis  for  a 
subset  of  factors  that  is  thought  to  be  most  influential  on  the  response.  Table  2 
provides  the  factors  and  their  levels  that  are  used  in  the  experimental  design. 

In  addition  to  the  factors  varied  in  the  design,  there  are  also  13  dependent 
input  variables  that  are  set  as  a  function  of  some  of  the  factors.  MANA  allows 
users  to  choose  two  modes  (advanced  and  cookie  cutter)  for  the  sensors.  While 
modeling  UAVs  in  the  model,  “advanced  mode”  is  used  for  their  sensors. 
Therefore,  three  levels  of  ranges,  time  between  detection,  and  their 
corresponding  probabilities  are  defined  rather  than  using  only  one  level.  Medium 
and  minimum  classification  ranges  are  calculated  by  multiplying  maximum  range, 
which  is  explicitly  varied  in  the  design,  by  3/5  and  1/5.  Similarly,  medium  and 
minimum  range  classification  probabilities  are  calculated  by  multiplying  maximum 
range  classification  probability  by  2/3  and  1/3.  The  time  between  detections  for 
medium  and  minimum  levels  are  set  by  multiplying  maximum  time  between 
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detections  by  2/3  and  1/3.  Finally,  fuel  usage  rate  in  refueling  state  is  computed 
by  multiplying  initial  fuel  level  by  1/5.  The  list  of  dependent  variables  which  form 
design  points  with  factors  varied  into  the  design  (independent  variables)  is 
provided  in  Appendix  A. 
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Table  2.  Factors  that  are  varied  in  the  experimental  design  and  their  ranges. 


UAV 

Factor  Name 

High 

Level 

Low 

Level 

Unit 

Type  of 
Variable 

Speed  in  Default  State 

450 

90 

km/hr. 

continuous 

Speed  in  Enemy  Contact  State 

300 

60 

km/hr. 

continuous 

Height  in  Default  State 

5000 

2000 

M 

continuous 

Height  in  Enemy  Contact  State 

4000 

1000 

M 

continuous 

Classification  Range  in  Default 

State  (Max.) 

10000 

1000 

m 

continuous 

Classification  Range  in  Enemy 

Contact  State  (Max.) 

10000 

1000 

m 

continuous 

Probability  of  Classification  at 

Max.  Range  in  Default  State 

0.264 

0.04 

continuous 

Probability  of  Classification  at 

Max.  Range  in  Enemy  Contact  State 

0.33 

0.05 

continuous 

Time  Between  Detection  at  Default 

State  (Max.) 

180 

15 

seconds 

continuous 

Time  Between  Detection  at  Enemy 

Contact  State  (Max.) 

120 

15 

seconds 

continuous 

Fuel  Level 

21600 

2700 

seconds 

continuous 

Time  Step  in  Refueling  State 

7200 

1800 

unit 

continuous 

1440 

360 

Time  steps 

Slew  Rate  in  Enemy  Contact  State 

90 

20 

degrees 
(per  second) 

continuous 

Scout 

Sensor  Range 

3000 

600 

m 

continuous 

Communication  Range  With 

Red  Team 

5000 

500 

m 

continuous 

Communication  Latency  With 

Red  Team 

20 

6 

Time  Step 

continuous 

Red 

Team 

Sensor  Range 

3000 

200 

m 

continuous 

Intra-Squad  Communication  Delays 

12 

6 

Time  Step 

continuous 

Civilian 

Communication  Link  Exist 

1 

0 

(Bernoulli) 

categorical 

Communication  Range  With 

Red  Team 

10000 

1000 

m 

continuous 

Communication  Latency  With 

Red  Team 

36 

12 

Time  Step 

continuous 
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1. 


Controllable  Factors 


Factors  can  be  classified  as  either  controllable  or  uncontrollable. 
Controllable  factors  can  be  controlled  by  the  systems  developers  or  operators — 
such  as  the  number  of  UAVs  used.  They  are  easy  to  handle  and  investigate.  In 
our  study,  controllable  factors  are  the  Unmanned  Aerial  Vehicle  (UAV) 
parameters  that  might  affect  the  UAVs’  detection  and  classification  abilities.  The 
list  of  factors  associated  with  UAVs,  which  are  given  in  Table  2,  constitutes  the 
controllable  factors  for  this  study.  Even  though  the  number  of  UAVs  is  not  directly 
included  in  the  design  sheet,  it  is  one  of  the  controllable  factors.  An  experimental 
design  is  developed  for  a  given  scenario,  and  the  whole  design  is  crossed  with 
the  number  of  UAVs  (1  up  to  4).  The  number  of  UAVs  assigned  over  the  area  of 
interest  affects  the  mean  coverage  ratio  over  time.  Similarly,  speed,  fuel  level, 
and  altitude  of  a  UAV  impact  on  the  area  covered  and  the  endurance.  Detection 
range,  time  between  detections,  slew  rate,  and  classification  probabilities  are  the 
factors  related  to  the  UAV’s  sensor(s).  The  factor,  time  step  in  refueling  state  is 
used  to  determine  how  many  time  steps  a  UAV  is  going  to  spend  in  their 
refueling  state  and  how  long  it  will  be  unavailable.  The  number  of  UAVs  and 
other  controllable  factors  provide  a  better  understanding  of  the  results.  How  does 
the  number  of  UAVs  affect  overall  performance?  Does  one  UAV  with  slower 
speed  perform  better  than  multiple  UAVs  with  higher  speeds?  Which 
configuration  of  UAV  provides  the  best  outcome  over  the  alternatives  analyzed? 

2.  Robust  Design 

Robust  design  is  a  method  pioneered  by  Dr.  Genichi  Taguchi  and  has 
been  broadly  used  to  improve  engineering  productivity  [36],  Robust  design 
focuses  on  uncontrollable  sources  of  variation,  which  may  exist  in  the  system  or 
the  environment,  as  well  as  mean  performance  of  the  system.  In  traditional 
experimental  design,  the  impact  of  uncontrollable  factors  on  the  response  is  often 
assumed  to  be  constant  during  the  experimental  runs.  But,  this  approach  forms  a 
restriction  on  real  world  decision  making  processes.  A  robust  design  optimizes 
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the  controllable  factors  in  the  presence  of  uncontrollable  factors  or  noise  factors. 
[37]  Therefore,  the  resulting  system  works  well  across  the  noise  factors  that  are 
included  in  the  experimental  design  [38], 

3.  Uncontrollable  Factors 

In  addition  to  factors  that  are  related  to  UAVs  and  directly  investigable, 
there  are  also  uncontrollable  factors  in  the  experimental  design.  Uncontrollable 
factors,  also  known  as  noise  factors,  are  factors  that  are  hard  to  regulate.  Yet, 
they  may  have  significant  effects  on  the  response.  Enemy  capabilities  are  a 
typical  example  of  the  uncontrollable  factors.  The  list  of  factors  associated  with 
Scouts,  Red  Teams,  and  Civilians,  which  are  given  in  Table  2,  constitutes  the 
uncontrollable  factors  for  this  study.  Red  Agents  have  a  perfect  stealth  value  in 
Enemy  Contact  3  state.  So,  all  the  factors  in  this  part  affect  their  ability  to  avoid 
detection. 

B.  NEARLY  ORTHOGONAL  LATIN  HYPERCUBE  (NOLH)  DESIGNS 

The  Nearly  Orthogonal  Latin  Hypercube  (NOLH)  design  developed  by 
Cioppa  (2002)  is  used  in  this  study  [39],  A  NOLH  design  is  a  special  case  of  Latin 
Hypercube  (LH)  designs.  The  experimental  region  in  a  NOLH  design  is  shaped 
by  the  minimum  and  maximum  levels  of  each  factor.  It  is  impossible  to  run  a 
model  for  all  possible  combinations  of  factor  levels,  even  with  today’s 
supercomputers.  If  there  are  many  factors  to  vary  in  the  design,  we  need 
advanced  techniques  to  analyze  the  effects  of  these  factors  simultaneously. 
Therefore,  a  NOLH  design  is  chosen  to  efficiently  provide  information  from  the 
experimental  runs  for  the  21  factors  in  this  study. 

The  Excel  spreadsheet,  from  the  SEED  Center  for  Data  Farming  at  the 

Naval  Postgraduate  School  in  Monterey,  was  developed  by  Professor  Susan 

Sanchez  and  is  used  to  build  the  NOLH  design  for  this  study  [29],  The 

spreadsheet  allows  building  different  designs  depending  on  the  number  of  factors 

(minimum  7,  maximum  29).  We  use  the  17-22  factors  design.  The  spreadsheet 

takes  the  high  level,  low  level,  and  decimal  points  as  inputs  and  returns  an 
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equally  spaced  permutation  of  values  in  between  defined  levels  for  each  factor. 
Figure  19  depicts  the  first  10  rows  of  the  NOLH  design  for  the  21  factors  varied  in 
this  study. 
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Figure  19.  NOLH  design  spreadsheet  for  17-22  factors  design  from  Tom 

Cioppa’s  2002  Ph.D.  dissertation. 


1.  Space-filling  Property  of  NOLH  Design 

The  NOLH  design  has  good  space-filling  filling  properties  [40],  Space¬ 
filling  means  the  design  points  (DPs)  are  scattered  uniformly  across  the  possible 
range  of  input  data.  The  scatter  plot  matrix  is  one  of  the  primary  graphical  tools 
used  to  visualize  relationships  between  variables.  Scatter  plot  matrices  for 
controllable  and  uncontrollable  factors  are  shown  in  Figure  20  and  Figure  21, 
respectively.  (Scatterplot  matrix  for  all  factors  is  included  in  Appendix  B.)  The 
entries  in  the  matrix  are  the  pairwise  comparisons  of  the  factors  in  the 
experimental  design.  For  example,  the  first  scatter  plot  of  Figure  20,  in  row  1, 
column  1 ,  contains  the  input  settings  between  UAV  speed  in  enemy  contact  state 
and  UAV  speed  in  default  state.  The  coverage  of  factor  levels  for  both  plots 
verifies  the  broad  space-filling  of  our  NOLH  design. 
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Scatterplot  Matrix 
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UAVSpeed  UAVSpeed_  UAVHeight  UAVHeight,  ange_Defa  nge_EnCon  ange_ENC  tMax_Defau  Max_EnCo  UAVFue  JAVTimeSte 

_Default  EnContad  .Default  ENContad  ult  tad  onatad  It  ntad  ILevel  InRefueling 


Figure  20.  Scatterplot  matrix  for  controllable  factors. 
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Scatterplot  Matrix 
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sorRange  d  d  orRange  ys  ComExist  Red 


Figure  21 .  Scatterplot  matrix  for  uncontrollable  factors. 


2.  Near  Orthogonality  of  NOLH  Design 

In  addition  to  the  space  filling  property,  orthogonality  is  also  desirable  in  a 
design  because  it  renders  uncorrelated  estimates  for  the  regression  coefficients. 
The  correlation  diagram  represented  in  Figure  22  confirms  that  the  correlations 
between  the  factors  in  the  experiment  are  insignificant.  A  value  of  1,  colored  red, 
implies  a  very  strong  positive  correlation,  while  a  value  of  -1,  colored  blue, 
represents  a  very  strong  negative  correlation.  The  diagonal  of  the  diagram 
displays  the  correlation  of  a  factor  by  itself. 
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Figure  22.  Correlation  diagram  for  all  of  the  factors  in  the  experiment. 


The  histogram  of  correlations,  among  all  factors,  given  in  Figure  23 
displays  that  the  mean  correlation  is  0.004.  The  results  of  the  correlation  diagram 
and  the  histogram  verify  that  our  design  is  nearly  orthogonal. 
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Figure  23.  Histogram  of  correlation  between  all  of  the  factors  in  the  experiment. 
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c. 


MODEL  RUNS 


1.  Tools  and  Techniques 

Sanchez’s  NOLH  design  spreadsheet  provides  129  DPs  for  21  factors 
varied  in  the  design.  After  using  the  design  sheet,  we  add  another  13  dependent 
input  variables  to  the  design.  The  whole  design  is  then  crossed  with  the  number 
of  UAVs  (1  up  to  4)  in  Excel.  The  final  design  we  create  contains  516  DPs. 

The  software  package  Xstudy,  written  by  SEED  Center  Research 
Associate  Steve  Upton,  is  used  to  create  the  mapping  between  variables  in  the 
design  and  elements  of  the  MANA  scenario  file,  and  to  specify  the  number  of 
independent  replications  per  DP.  Consequently,  a  design  file  in  xml  format  is 
generated.  The  XStudy  software  package  is  available  at  http://harvest.nps.edu.  A 
follow  on  to  XStudy  is  software  called  OldMcData,  which  is  also  written  by  Steve 
Upton,  it  manages  the  generation  of  scenario  files  for  each  design  point  using  the 
experimental  design  sheet,  the  baseline  MANA  xml  files  created  for  each 
scenario,  and  the  XStudy  xml  files.  The  OldMcData  software  package  is  also 
available  at  http://harvest.nps.edu. 

The  open  source  software  package  Condor,  which  is  available  from  the 
University  of  Wisconsin  at  http://www.cs.wisc.edu/condor/,  distributes  and 
manages  the  jobs  across  a  set  of  available  processors.  In  our  study,  the  model  is 
run  on  cluster,  called  “reaper,”  with  44  processors.  Finally,  a  postprocessor 
included  in  OldMcData  is  used  to  combine  all  of  the  simulation  outputs,  with  their 
corresponding  input  values,  into  a  single  comma-separated  (csv)  file. 

2.  Time  Analysis 

The  run  time  for  a  single  replication  of  the  base  case  is  about  7  to  8 

minutes  on  average.  The  number  of  replications  desired  is  calculated  using  the 

formula  in  equation  1.  We  use  an  a  value  of  0.05  and  /3  value  of  0.10.  Therefore, 

we  need  to  take  about  200  samples  in  order  to  have  a  90%  chance  of  detecting  a 

difference  in  means  of  1  for  a  test  with  a  5%  type-1  error  rate.  As  a  result,  200 

replications  are  made  for  each  design  point.  Figure  24  displays  the  change  in 
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sample  size  required  based  on  the  empirical  variance  of  mean  number  of  Red 
Agents  classified. 
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Figure  24.  Number  of  replications  required  per  design  point. 


We  end  up  with  a  total  of  103,200  simulated  terrorist  incursions,  which 
would  have  taken  about  573  days  to  run  using  only  a  single  processor.  However, 
since  the  model  is  run  on  a  cluster  with  44  processors,  it  takes  about  only  about 
13  days  to  complete  all  of  the  runs.  The  model  runs  took  longer  time  than 
expected  due  to  failures  on  a  section  of  node  during  the  runs.  However,  this 
analysis  would  require  approximately  35*  1011  replications  with  a  traditional, 
brute-force  full-factorial  design.  Graphical  comparisons  for  the  required  number 
of  runs  and  time  needed  to  complete  these  runs  are  provided  in  Figure  25  and 
Figure  26,  respectively. 
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Number  of  Runs  Required  for  Analysis 
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Figure  25.  Graphical  comparison  of  the  number  of  replications  required  for  the 
study  with  full  factorial  design,  fractional  factorial  design,  and  NOLH 

design. 
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Figure  26.  Graphical  comparison  of  the  years  required  to  complete  number  of 
runs  with  full  factorial  design,  fractional  factorial  design,  and  NOLH 

design. 


52 


V.  DATA  ANALYSIS 


In  this  chapter,  we  provide  a  quick  review  of  the  analysis  tools  utilized  in 
this  study.  Then,  we  discuss  the  results  of  some  initial  runs  that  were  made 
without  using  an  experimental  design.  In  the  last  section,  we  present  a 
discussion  of  various  statistical  modeling  techniques  to  analyze  the  relationships 
between  the  factors  in  the  experimental  design  and  our  MOEs. 

A.  ANALYSIS  TOOLS 

1.  JMP 

JMP,  one  of  the  two  tools  used  to  analyze  the  data,  was  created  by  SAS 
in  1986  as  a  data  exploration  tool.  According  to  JMP’s  official  website,  available 
at  http://www.jmp.com,  it  provides  graphical  and  visual  interpretations  of  data 

[41] .  Additionally,  JMP  has  a  dynamic  user  interface  that  allows  users  to  conduct 
interactive  data  analysis. 

JMP  enables  a  wide  variety  of  statistical  techniques,  such  as  linear 
regression,  nonlinear  regression,  partition  trees,  time  series  analysis,  Gaussian 
process,  neural  networks,  and  more.  We  use  JMP  Statistical  Discovery  Software 
version  10.0.0  to  perform  the  regression  analysis  for  this  study. 

2.  R 

R  is  a  language  and  environment  for  data  manipulation,  statistical 
computing,  and  graphical  display.  It  was  developed  by  John  Chambers  and  his 
colleagues  based  on  the  code  written  for  the  S  language  and  its  environment 

[42] ,  R  is  free  software  available  in  source  code  format  and  capable  of  working 
on  various  platforms,  such  as  Windows,  UNIX,  and  MacOS.  Users  can  extend 
R’s  functionality  by  adding  new  functions  or  installing  new  packages.  In  addition 
to  JMP,  we  use  R  as  a  supplementary  tool  for  our  analysis. 
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B.  RESULTS  OF  INITIAL  RUNS 


Following  the  creation  of  the  model  in  MANA,  200  initial  runs  are  made  for 
the  base  case  for  each  scenario.  We  expect  to  see  how  results  change 
depending  on  the  number  of  UAVs  through  these  preliminary  runs  and  get 
information  on  the  variability  associated  with  the  numerous  random  elements  in 
the  model. 

1 .  Total  Number  of  Red  Agents  Arrived  at  Destination 

There  are  150  Red  Agents  that  are  trying  to  cross  the  border  in  the 
simulation  model.  They  can  either  be  detected  and  classified  by  Blue  Agents,  or 
they  can  reach  their  destination  without  being  detected.  Upon  reaching  its  goal 
undetected,  each  Red  Agent  kills  one  Blue  Target  which  is  placed  at  the 
destination  of  Red  Agents.  Thus,  the  number  of  killed  Blue  Targets  forms  a 
surrogate  for  the  number  of  Red  Agents  that  reach  the  destination.  Figure  27 
displays  the  distribution  of  the  number  of  Red  Agents  that  reach  the  destination 
for  the  scenarios  analyzed.  The  histogram  provides  the  distribution  of  arrived 
Red  Agents  within  the  predefined  time  frame.  In  addition  to  the  histogram,  box 
plots  depict  the  summary  statistics  of  the  data  pictorially.  According  to  the 
summary  statistics,  on  average,  60  of  the  Red  Agents  are  able  to  reach  their 
destination  within  the  predefined  maximum  timeline.  The  mean  number  of  arrived 
Red  Agents  decreases  with  the  number  of  UAVs  (0  up  to  4).  The  mean  number 
of  arrived  Red  Agents  drops  to  1 .21  for  scenario  five  with  four  UAVs. 
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Distributions 


AllegICas.Blue.NoUAV 


AllegICas.Blue.OneUAV  AllegICas.Blue.TwoUAV  AllegICas.Blue.ThreeUAV  AllegICas.Blue.FourUAV 
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69 
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67 

90.0% 

64 

75.0% 
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62 

50.0% 

median 

60 

25.0% 

quartile 

58 

10.0% 

55 

2.5% 

53 

0.5% 

51.005 

0.0% 
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51 

Summary  Statistics 

Mean  59.695 

Std  Dev  3.5883641 

Std  Err  Mean  0.2537357 
Upper  95%  Mean  60.195356 
Lower  95%  Mean  59.194644 
N  200 
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10.0% 

5 
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3 

0.5% 

1.005 

0.0% 
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1 

Summary  Statistics 

Mean  26.645 

Std  Dev  16.369139 

Std  Err  Mean  1.1574729 
Upper  95%  Mean  28.927486 
Lower  95%  Mean  24.362514 
N  200 
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51 
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35 
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9 
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5 
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2 
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0 

2.5% 

0 
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0 

0.0% 
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0 

Summary  Statistics 

Mean  10.37 

Std  Dev  12.989486 

Std  Err  Mean  0.9184953 
Upper  95%  Mean  12.181233 
Lower  95%  Mean  8.5587671 
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0 

0.5% 
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Mean  3.615 
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Std  Err  Mean  0.6863595 
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90.0% 
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75.0% 

quartile 
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50.0% 

median 
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quartile 

0 

10.0% 

0 

2.5% 
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0.5% 

0 
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minimum 

0 

Summary  Statistics 
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N  200 


Figure  27.  Distribution  of  total  number  of  Red  Agents  that  reach  the  destination. 


2.  Total  Number  of  Classified  Red  Agents 

The  Red  Agents  in  the  simulation  model  can  only  be  classified  by  the 
UAVs.  The  Red  Agents  are  the  most  valuable  targets.  Therefore,  they  have  the 
highest  priority  in  the  UAV  sensor  set  up.  Following  their  classification,  Red 
Agents  are  devolved  to  the  Blue  Killing  Agent  to  prevent  multiple  classifications. 
The  distribution  of  the  number  of  classified  Red  Agents  is  provided  in  Figure  28. 
The  number  of  classified  Red  Agents  is  0,  as  expected,  when  there  are  no  UAVs 
over  the  region.  The  mean  number  of  classified  Red  Agents  increases  with  the 
number  of  UAVs.  The  mean  of  the  total  number  of  classified  Red  Agents  jumps 
from  0  to  122.485  for  scenario  two  with  one  UAV.  The  total  number  of  classified 
Red  Agents  can  reach  150  for  scenario  three  and  scenario  four. 
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Distributions 

Alleg2Cas.Red.NoUAV 


Quantiles 
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Figure  28.  Distribution  of  total  number  of  classified  Red  Agents. 


The  average  number  of  classified  Red  Agents  is  plausibly  high  even  with 


two  UAVs  over  the  area  of  interest.  Nevertheless,  the  variability  decreases  when 
we  increase  the  number  of  UAVs.  Figure  29  and  Figure  30  provide  the  average 


and  standard  deviation  of  the  number  of  arrived  and  classified  Red  Agents  for  the 
results  of  200  initial  runs  in  each  scenario.  As  a  result  of  the  intuition  gained 
through  the  initial  runs,  the  design  is  crossed  for  the  number  of  UAVs  (1  up  to  4). 
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Average  Number  of  Arrived  and 
Classified  Red  Agents 


Number  of  UAVs 


■  Arrived  Red  Agents 

■  Classified  Red  Agents 


Figure  29.  Average  number  of  arrived  and  classified  Red  Agents  based  on  the 

results  of  200  initial  runs. 


Standard  Deviation  of  Arrived  and 
Classified  Red  Agents 


■  Arrived  Red  Agents 

■  Classified  Red  Agents 


Figure  30.  Standard  deviation  of  arrived  and  classified  Red  Agents  based  on  the 

results  of  200  initial  runs. 
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C.  RESULTS  OF  RUNS  FOR  CROSSED  DESIGN 

We  use  the  mean  of  200  replications  as  our  measure  of  effectiveness 
(MOEs).  As  a  result,  we  reduce  the  number  of  data  points  from  103,200  to  516. 
This  facilitates  visual  displays  of  the  data. 

1 .  Measure  of  Effectiveness 

The  purpose  of  the  study  is  to  analyze  the  effectiveness  of  UAVs  on 
detection  of  terrorists  over  a  hard  topographical  region.  Therefore,  we  describe 
two  MOEs  to  measure  the  effect  of  model  factors  on  UAV  performance.  The 
mean  numbers  of  arrived  and  classified  Red  Agents  are  the  two  MOEs 
investigated  for  the  purpose  of  this  analysis.  An  initial  assessment  of  the  MOEs 
highlighted  that  there  is  a  strong  negative  correlation  between  the  numbers  of 
arrived  and  classified  Red  Agents.  Figure  31  shows  the  correlation  between 
these  two  MOEs.  The  correlation  between  the  MOEs  is  -0.6898. 
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Multivariate 

Correlations 

Mean  (#of_Arri  ve  d_Re  dAg  ents)  Mean  (#of_C  I  as  s  if  i  e  d_Re  dAgent) 
Mean(#of_Arrived_RedAgents)  1.0000  -0.6898 

Mean(#of_Classified_RedAgent)  -0.6898  1.0000 


Scatterplot  Matrix 


Figure  31 .  Scatterplot  of  the  mean  numbers  of  arrived  and  classified  Red  Agents. 

Since  the  two  MOEs  are  so  highly  correlated,  we  choose  the  number  of 
classified  Red  Agents  as  the  primary  MOE  to  study  the  effect  of  factors  on  the 
UAV’s  performance.  The  color  map  of  correlations,  ranging  from  -1  to  1  and 
provided  in  Figure  32,  shows  that  the  two  MOEs  have  a  strong  negative 
relationship.  Consequently,  we  expect  independent  regressors  to  have  reverse 
impacts  on  these  two  MOEs. 
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2.  T-test  for  Comparison  of  the  Sample  Mean 

The  scenarios  developed  for  the  different  number  of  UAVs  are  compared 
with  each  other,  in  terms  of  mean  number  of  classified  Red  Agents,  to  explore 
how  different  number  of  UAVs  contributes  to  detection  and  classification  of  Red 
Agents. 

The  results  of  the  T-Test  for  the  mean  number  of  classified  Red  Agents 
yield  that  there  is  a  statistically  significant  difference  between  the  scenarios  at  a 
95%  confidence  level  (or  when  a  =  .005).  As  expected,  the  mean  number  of 
classified  Red  Agents  increases  with  the  number  of  UAVs  as  shown  in  Figure  33. 
A  detailed  comparison  report  of  each  pair  is  provided  in  Appendix  C. 

In  addition  to  the  T-test,  we  look  at  the  trend  between  number  of  UAVs 
and  corresponding  coefficient  estimates,  and  verify  the  linear  relationship.  Then, 
we  fit  two  distinct  models  changing  the  classification  of  number  of  UAVs  as 
continuous  and  as  discrete.  The  F-test  critical  value,  calculated  based  on  these 
two  models  is  0.47,  which  is  much  less  than  the  F-statistic  critical  value  of  2.19 
[43],  Hence,  we  fail  to  reject  the  null  hypothesis  and  conclude  that  it  is  okay  to 
treat  the  number  of  UAVs  as  a  continuous  variable.  As  a  result,  we  conduct  the 
rest  of  the  analysis  using  the  number  of  UAVs  as  a  continuous  variable. 
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Means  Comparisons 

Comparisons  for  each  pair  using  Student's  t 


Ordered  Differences  Report 

Level  -Level 

Difference 

Std  ErrDif 

Lower  CL 

Upper  CL 

p-Value 

4AOR  1AOR 

23.12419 

3.029404 

17.17259 

29.07578 

<0001* 

1  ur 

3AOR  1A0R 

16.46512 

3.029404 

10.51352 

22.41671 

<0001* 

1 :  :  : 

4AOR  2AOR 

14.82438 

3.029404 

8.87279 

20.77597 

<0001* 

1  :  A?  : 

2AOR  1A0R 

8.29981 

3.029404 

2.34821 

14.25140 

0.0064* 

3AOR  2A0R 

8.16531 

3.029404 

2.21372 

14.11690 

0.0073* 

4AOR  3A0R 

6.65907 

3.029404 

0.70748 

12.61066 

0.0284* 

ZL 

Figure  33.  T-test  for  comparison  of  mean  number  of  classified  Red  Agents. 


3.  Linear  Regression 


We  use  linear  regression  to  explore  the  relationship  between  the  response 
variable  and  the  input  factors.  The  linear  model  presumes  that  the  regression 
function  is  linear  or  that  the  linear  model  is  an  acceptable  approximation  [44], 
The  basic  mathematical  interpretation  of  the  linear  model  is  stated  by  the 
following  equation  [44]  : 


RX) 


P  \ 


\  i= i  J 


(2) 


In  the  equation,  xj  represents  the  independent  regressors.  The  factors 


presented  in  Table  2  are  the  independent  regressors  of  our  study.  J30  represents 


the  intercept  and  jBy  stands  for  the  coefficients  of  the  independent  regressors.  In 
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our  study,  we  want  to  fit  the  number  of  classified  Red  Agents  using  the 
regression  function  on  the  factors. 

a.  The  Basic  Model 

The  mean  number  of  classified  Red  Agents  is  used  as  the 
response  variable  to  conduct  a  stepwise  linear  regression  [44]  with  all  of  the 
independent  factors  in  the  experimental  design.  The  resulting  model  has  an  R2 
value,  the  coefficient  of  determination,  of  0.7948  and  an  adjusted  R2  value  of 
0.7903,  which  means  that  the  regressors  in  the  model  explain  approximately 
80%  of  the  variability  in  the  response.  We  then  perform  model  diagnostics  using 
the  residual  plots  shown  in  Figure  34. 
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Figure  34.  Residuals  plots  for  the  stepwise  regression  model. 


All  Cook’s  distance  values  are  less  than  one,  which  shows  that 
there  are  no  influential  points.  However,  the  residual  versus  fitted  plot  indicates 
non  constant  variance  in  the  model.  Non  constant  variance,  or 
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heteroscedasticity,  is  anticipated  in  the  simulation  model  due  to  the  hard 
topography  of  the  region.  Additionally,  the  normal  quantile  plot  points  out  that  the 
errors  are  not  normally  distributed.  After  the  diagnosis  of  the  initial  model,  a  Box- 
Cox  transformation  is  performed  on  the  response  variable.  We  observe  a  lambda 
value  of  0.4  through  the  plot  provided  in  Figure  35  and  conduct  a  power 
transformation  on  response. 


-2-1012 


k 

Figure  35.  Graphical  result  of  Box-Cox  transformation  suggests  a  power 

transformation  of  0.4  on  the  response. 

We  fit  a  new  model  using  the  response  variable  with  the  power 
transformation.  The  heteroscedasticity  and  normality  problems  seem  to  be 
handled  by  this  according  to  the  plots  given  in  Figure  36.  Moreover,  the  R2  value 
increases  from  0.7923  to  0.8837.  Finally,  we  conduct  a  Durbin  Watson  test  to 
ensure  that  the  autocorrelation  between  the  regressors  is  zero.  The  test  returns  a 
Durbin  Watson  statistic  of  2.0683  with  a  p-value  of  0.7663,  which  is  greater  than 
our  significance  level  of  0.05  (a  =  0.05).  Therefore,  there  is  no  strong  evidence 
that  violates  the  null  hypothesis  and  we  conclude  that  the  autocorrelation  is  zero. 
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Figure  36.  Residuals  plot  after  the  Box-Cox  transformation  on  the  response. 


The  p-value  of  the  basic  model  is  less  than  0.0001,  which  indicates 
there  is  a  highly  significant  relationship  between  the  response  and  regressors 
included  in  the  model.  Figure  37  provides  the  summary  statistics  for  the  basic 
model. 
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Actual  by  Predicted  Plot 


Summary  of  Fit  [  [Analysis  of  Variance 
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Figure  37.  The  statistical  results  for  the  basic  model  after  the  Box-Cox 

transformation. 


There  are  1 1  factors  that  have  the  highest  impact  on  the  response 
when  a  =  0.05.  Table  3  provides  the  list  of  these  factors  in  the  order  of  their 
significance  level.  The  t-ratio  changes  with  the  contribution  of  the  factor  to  the 
explanatory  power  of  model.  Therefore,  the  factors  with  the  highest  t-ratio  are  the 
most  significant  in  the  model.  UAV  classification  range  in  the  default  state  and 
probability  of  classification  in  the  enemy  contact  state,  Red  Team  sensor  range, 
number  of  UAVs,  time  between  detections,  and  UAV  fuel  level  are  the  major 
factors  that  affect  the  classified  number  of  Red  Agents  within  the  predefined  time 
frame. 
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Table  3.  Sorted  parameter  estimates  for  the  basic  model 


Sorted  Parameter  Estimates 

Term 

Estimate 

U  AVM  axC  1  a  s  s  Ra  n  g  e_Def  ault 

0.0003713 

Red_SensorRange 

-0.000776 

UAVPCIassAtMaxRange_EnContact 

5.7812371 

NumberofUAVs 

0.4190199 

U  AVTi  m  eBtwD  etM  ax_Def  ault 

-0.007962 

UAVFuelLevel 

2.8363e-5 

U  AVTi  m  e  BtwD  etM  ax_E  n  C  ontad 

-0.004233 

UAVTimeStepInRefueling 

-0.000328 

Civilian_ComExist 

-0.192873 

UAVSpeed_Default 

-0.000758 

Civilian_ComRangeWifiRed 

-2.538e-5 

Std  Error  t  Ratio 

Prob>|t| 

8.816e-6  42.11 

<  0001* 

2.864e-5  -27.11 

1 

<  0001* 

0.286438  20.18 

<  0001* 

0.020866  20.08 

<  0001* 

0.000484  -16.46 

d 

<  0001* 

4.225e-6  6.71 

!  i 

<  0001* 

0.000748  -5.66 

:C 

:  :  :  : 

<  0001* 

7.44e-5  -4.41 

<  0001* 

0.047131  -4.09 

<  0001* 

0.000223  -3.40 

0.0007* 

8.788e-6  -2.89 

0.0040* 

b.  The  Saturated  Model 

Following  the  basic  model,  we  fit  models  that  include  interactions 
and  higher  order  terms.  First,  we  fit  a  model  with  all  pairwise  interactions  among 
the  significant  factors  and  compare  this  model  with  the  basic  model,  without 
interaction  effects,  using  an  F  test.  The  F  test  returns  a  p-value  of  2.2e-16.  As  a 
result,  we  strongly  reject  the  null  hypothesis  and  conclude  that  the  saturated 
model  provides  an  improvement  over  the  basic  model.  The  model  with  all  two 
way  interactions  has  an  R2  value  of  0.9334  with  30  terms  included  in  the  model. 
The  statistical  results  and  sorted  parameter  estimates  for  the  model  with 
interaction  effects  are  shown  in  Appendix  D. 

Likewise,  we  fit  a  model  including  quadratic  effects  of  significant 
factors  in  the  model  using  stepwise  linear  regression.  The  results  indicate  that 
quadratic  term  for  time  between  detections,  UAV  classification  range,  number  of 
UAVs,  Red  Team  sensor  range,  and  UAV  fuel  level  are  significant.  An  improved 
R2  value  of  0.9684  is  provided  by  the  model  with  these  significant  interactions 
and  quadratic  terms.  Figure  38  provides  the  statistical  summary  of  the  resulting 
model  with  28  terms  contributing  to  the  explanatory  power  of  the  model. 
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Figure  38.  Statistical  results  for  the  model  with  interactions  and  quadratic  terms. 


Finally,  we  observe  the  change  in  the  R2  value  of  the  model  by 
adding  parameters  to  the  model  in  their  order  of  their  significance.  The  purpose 
is  to  reduce  the  complexity  without  significantly  demoting  the  predictive  capability 
of  the  model.  That  is,  we  desire  a  parsimonious  model.  According  to  the  plot  in 
Figure  39,  the  R2  value  almost  evens  out  after  the  seventeenth  parameter.  This 
indicates  that  only  17  of  the  28  parameters  in  the  model  with  main  effects, 
interactions,  and  quadratic  effects  contribute  the  vast  majority  of  the  predictive 
power  of  the  model. 
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Figure  39.  R2  value  plot  for  the  number  of  parameters  included  in  the  model  in 

the  order  of  their  significance. 


The  reduced  size  model,  which  has  an  R2  value  of  0.957048, 
includes  the  factors  listed  in  Table  4.  We  see  that  UAV  classification  range  in 
default  state  and  probability  of  classification  in  enemy  contact  state,  Red  sensor 
range,  number  of  UAVs,  time  between  detections  in  the  default  state,  and 
quadratic  effects  of  classification  range  have  the  highest  impact  on  the  model. 


Table  4.  Sorted  parameter  estimates  for  the  reduced-size  model  highlighted 

by  the  R2  method  given  in  Figure  39. 
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4.  Lasso  Regression 

The  least  absolute  shrinkage  and  selection  operator  (lasso)  is  a  method 
for  estimation  in  linear  models.  It  drops  some  coefficients  off  and  sets  the  others 
to  0,  thus  attempting  to  return  a  less  complex  model  [45],  The  lasso  method  uses 
a  diamond  shaped  constraint  region  and  performs  LI  shrinkage.  The  sum  of 
squares  is  calculated  based  on  various  coefficients  that  are  drawn  as  red  contour 
lines  in  Figure  40.  If  the  sum  of  squares  hits  one  of  the  corners  of  blue  diamond, 
then  corresponding  coefficients  are  set  to  0  [46],  The  amount  of  shrinkage  in  the 
model  is  driven  by  a  tuning  parameter  lambda. 


Figure  40.  Estimation  picture  for  the  lasso  regression  where  ellipses  are  the 
contours  of  the  least  squares  error  function  and  blue  area  is  the 
constraint  region.  (From:  onlinecourses.science.psu.edu). 

We  apply  this  method  to  the  saturated  model  with  28  terms.  Figure  41 
displays  the  cross-validated  Mean  Square  Error  (MSE)  plot.  The  dashed  line  in 
the  plot  represents  the  lambda  value  with  minimal  MSE  plus  one  standard 
deviation.  As  lambda  increases,  MSE  also  increases  rapidly.  This  means  that  the 
coefficients  are  reduced  too  much  and  the  resulting  model  has  a  poor 
explanatory  power.  As  lambda  decreases,  a  smaller  number  of  coefficients  are 
set  to  0. 
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Figure  41 .  The  cross-validated  Mean  Square  Error. 


Then,  we  choose  the  least  complex  model  using  the  “one-standard-error” 
rule.  The  trace  plot  of  coefficients  is  provided  in  Figure  42.  The  plot  shows  the 
nonzero  coefficients  in  the  model  for  different  values  of  lambda.  The  dashed  line 
in  the  plot  represents  the  lambda  value  corresponding  to  minimal  MSE  and 
number  of  terms  included  in  the  model  within  one  standard  error.  The  lasso 
asserts  17  terms  based  on  the  minimum  least  squares  error  approach.  The 
factors  chosen  by  lasso  regression  are  almost  same  with  the  factors  in  Table  4 
except  for  the  quadratic  terms.  Different  than  quadratic  effects,  lasso  regression 
keeps  civilian  communication  range  and  it’s  interactions  with  the  factors  time 
between  detections  and  civilian  communication  exists. 
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Figure  42.  Trace  plot  of  coefficients  fit  by  lasso. 


5.  Regression  Tree  Analysis 

Regression  trees,  also  known  as  partition  trees,  provide  another  method 
to  analyze  regression  problems.  A  regression  tree  represents  the  decision 
making  steps  visually  and  iteratively.  The  tree  model  is  attained  by  recursively 
partitioning  the  data  space  and  fitting  a  simple  model  based  on  each  partition. 
Then,  the  results  can  be  displayed  as  a  regression  tree  [47],  The  determination 
of  the  optimal  split  point  is  made  based  on  the  minimum  sum  of  squares  [44], 
After  each  split,  the  succeeding  ideal  split  is  determined  within  each  partition. 

We  build  a  regression  tree  for  the  mean  number  classified  Red  Agents 
given  all  input  factors.  When  we  look  at  the  first  4  splits  presented  in  Figure  43, 
we  see  that  the  results  conform  to  the  results  of  stepwise  regression  and  lasso 
regression  in  previous  sections.  Each  box  in  the  tree  structure  provides  the  factor 
and  level  of  optimal  split.  In  addition  to  factor  and  level,  the  number  of  data  points 
included  in  the  split,  mean  number  of  classified  Red  Agents,  and  standard 
deviation  within  the  split  are  also  provided  in  the  box.  We  see  that  the  first  split  is 
on  UAV  default  state  classification  range  of  5000  meters.  200  observations  have 
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classification  range  less  than  5000  meters  with  a  mean  response  value  of  1 1 .23, 
whereas  316  observations  have  classification  range  greater  than  5000  meters 
with  a  mean  response  value  of  38.56.  The  next  split  occurs  for  Red  Agent  sensor 
range  of  1400  meters.  The  mean  number  of  classified  Red  Agents  is  52.79  when 
the  Red  sensor  range  is  less  than  1400  meters. 


Figure  43.  Partition  tree  model  for  the  number  of  classified  Red  Agents. 


Following  the  initial  splits,  we  grow  a  regression  tree  with  20  splits  and 
observe  an  R2  value  of  0.826  (The  tree  structure  for  first  13  splits  is  provided  in 
Appendix  E).  Figure  44  displays  the  improvement  of  R2  value  with  number  of 
splits. 
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Figure  44.  Split  history  of  tree  model. 


The  variable  contributions  to  the  explanatory  power  of  the  tree  model  are 
provided  in  Figure  45.  The  tree  model  indicates  that  we  can  explain  variability  in 
the  response  with  a  less  complex  model  including  just  seven  factors.  Following 
the  tree  model,  we  fit  a  linear  regression  model  with  these  highlighted  seven 
factors  and  observe  an  R2  value  of  0.87.  The  results  of  the  tree  model  yield  that 
UAV  classification  performance  is  mainly  driven  by  the  classification  range  in  the 
default  state,  but  changes  depending  on  the  Red  Team  sensor  range,  UAV 
classification  probability,  number  of  UAVs,  time  between  detections  in  default 
state,  UAV  speed,  and  UAV  fuel  level. 


Column  Contributions 

Number 

Term  of  Splits  SS 

UAVMaxClassRange_Default  4  489.52044 

Red_SensorRange  3  194.34537 

UAVPCIassAtMaxRange_EnContad  4  141.99616 

NumberofUAVs  3  85.56303 

UAVTimeBtwDetMax_Default  3  42.72079 

UAVSpeed_Default  1  36.90650 

UAVFuelLevel  1  10.52682 


Figure  45.  Variable  contributions  to  the  explanatory  power  of  the  tree  model. 
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6.  Random  Forest  Analysis 

Random  forests,  first  introduced  by  Leo  Breiman,  are  “black-box” 
supervised  learning  methods  for  classification  and  regression.  Thus,  their 
interpretation  capability  is  not  as  powerful  as  their  prediction  power.  Random 
forests  grow  many  classification  trees,  and  these  trees  vote  the  most  popular 
class  for  a  given  input  set  [48],  The  number  of  trees  and  number  of  variables  to 
try  in  each  split  can  be  determined  by  users.  If  we  assume  that  a  number  of  m 
variables  can  be  used  in  each  split,  the  algorithm  randomly  selects  m  variables 
out  of  all  available  candidates.  Then,  the  best  split  is  chosen  over  the  m 
variables.  Random  forests  grow  the  largest  possible  tree  without  pruning.  We 
grow  500  trees  and  subset  4  variables  to  try  in  each  split.  The  random  forests 
model  explains  94.36%  of  the  variability  in  the  response  with  a  MSE  of  0.1331. 

Random  forests  can  also  be  used  to  determine  variable  importance  in  a 
regression  problem.  The  variable  importance,  based  on  a  decrease  in 
classification  accuracy,  pinpointed  by  the  random  forests  model,  can  be  seen  in 
Figure  46.  Similar  to  results  of  regression  and  tree  analysis;  random  forests  also 
highlight  seven  factors.  We  again  see  that  UAV  classification  range  in  default 
state  and  Red  Team  sensor  range  have  the  highest  impact  on  UAV  performance 
in  terms  of  detection  and  classification  of  Red  Agents. 
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Figure  46.  Sorted  variable  importance  pinpointed  by  the  random  forests  model. 


75 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


76 


VI.  CONCLUSION 


The  main  purpose  of  this  study  is  to  explore  the  effectiveness  of 
Unmanned  Aerial  Vehicles  (UAVs)  in  helping  secure  a  border  characterized  by 
rough  terrain  and  active  terrorists.  Additionally,  we  investigated  how  the  number 
of  UAVs  and  performance  characteristics  contribute  to  border  security. 

We  have  identified  some  important  factors  related  with  UAVs  and 
infiltration  operations  based  on  our  scenarios.  It  would  be  almost  impossible  to 
model  every  aspect  of  an  infiltration  scenario.  Yet,  this  was  not  the  purpose  of 
our  study.  We  focus  our  study  on  gross  level  effects  of  factors  of  interest  to  look 
at  if  it  is  possible  to  use  UAVs  for  intelligence,  surveillance,  and  reconnaissance 
purposes  over  a  hard  topographical  region. 

We  utilize  four  analysis  techniques  to  study  the  mean  number  of  classified 
Red  Agents  (terrorists):  stepwise  linear  regression,  lasso  regression,  regression 
trees,  and  random  forests  on  the  results  of  103,200  simulated  infiltration 
scenarios.  The  four  analysis  techniques  highlight  similar  factors  as  the  most 
important  factors  with  additional  insights. 

A.  PRIMARY  FINDINGS 

•  Agent-based  models  provide  us  a  modeling  platform  to  create  and  analyze 
scenarios  in  a  short  time  period. 

•  The  data  farming  process  is  a  powerful  technique  to  study  the  effects  of 
numerous  factors  simultaneously. 

•  The  use  of  UAVs  significantly  enhances  the  detection  and  classification  of 
terrorists  operating  or  transiting  in  the  region. 

•  The  classification  range  of  a  UAV  has  a  strong  positive  affect  in  the 
number  of  classified  terrorists.  First  split  of  regression  tree  analysis 
highlights  the  importance  of  having  a  classification  range  greater  than  5 
km. 

•  The  sensor  range  of  terrorists  to  detect  UAVs  has  a  negative  impact  on 
the  number  of  classified  terrorists.  Even  though  UAV  altitude  is  not 
highlighted  as  a  significant  factor,  assigning  UAVs  at  higher  altitudes  can 
reduce  their  probability  of  detection  by  terrorists. 
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•  The  classification  probability  of  a  UAV  is  also  important.  A  1%  increase  in 
classification  probability  results  in  more  classified  terrorists.  Additionally, 
regression  tree  analysis  yields  that  a  maximum  range  (>  5000  meters) 
classification  probability  greater  than  0.17  provides  better  results  in  the 
classification  of  terrorists.  From  a  practical  sense,  classification  probability 
of  a  camera,  which  is  onboard  the  UAV,  leads  to  the  number  of  effective 
looks  needed  to  search  an  area.  So,  this  is  an  important  factor  to  take  into 
account  in  terms  of  mission  plan  and  organization. 

•  We  cannot  talk  about  an  optimal  number  of  UAVs  to  assign  over  the  area 
of  interest,  but  regression  tree  analysis  indicates  that  assigning  three  or 
more  UAVs  results  in  more  classified  terrorists  with  a  lower  variance. 

•  The  time  between  detections  of  entities  is  another  important  factor  in 
terms  of  a  UAV’s  classification  capability.  A  one  second  increment  of  time 
between  detections  in  the  default  state  causes  a  larger  decrease  in  the 
number  of  classified  terrorists  than  a  one  second  increment  of  time 
between  detections  in  enemy  contact  state. 

•  Refueling  time  is  a  bigger  determinant  of  success  than  the  initial  fuel  level 
of  a  UAV.  Thus,  it  is  important  for  ground  bases  to  be  able  to  quickly  get 
the  UAVs  back  up. 

•  Increasing  the  UAV’s  speed  in  the  default  state  has  a  negative  impact  on 
UAV  classification  performance.  Regression  tree  analysis  shows  that  UAV 
speeds  higher  than  250  km/hr  noticeably  decreases  the  number  of 
classified  terrorists.  Therefore,  the  speed  of  flight  is  critical.  If  it  is  too  high, 
UAVs  will  quickly  occupy  images  of  the  terrain  and  will  miss  the  details. 
On  the  contrary,  if  the  speed  is  too  low,  then  there  will  be  less  variation  in 
the  occupied  image,  hence  less  information  will  be  gathered  through  the 
scans. 

•  The  existence  of  communication  links  between  the  civilians  and  terrorists 
is  another  significant  factor  on  UAV  performance.  The  existence  of  civilian 
communication  with  terrorist  caused  a  reduction  in  the  number  of 
classified  terrorists.  Therefore,  SIGINT  and  electronic  warfare  capabilities 
can  be  considered  in  addition  to  UAV  performance  characteristics. 

B.  ADDITIONAL  FINDINGS 

•  The  stepwise  linear  regression  indicates  a  significant  interaction  between 
classification  range  and  classification  probability.  Increasing  classification 
probability  has  a  greater  effect  on  the  number  of  classified  terrorists  when 
the  classification  range  is  high. 

•  Another  significant  interaction  highlighted  by  the  stepwise  linear 
regression  is  between  terrorists’  sensor  range  and  civilian  communication 
range  with  terrorists.  For  example,  when  the  terrorist  sensor  range  is  low, 
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more  advanced  civilian  communication  abilities  can  still  have  a  negative 
effect  on  the  performance  of  a  UAV. 

•  A  third  significant  interaction  emphasized  by  the  stepwise  linear 
regression  is  between  the  number  of  UAVs  and  UAV  classification  range. 
When  the  number  of  UAVs  decreases,  the  number  of  classified  terrorists 
is  greater  for  increasing  classification  range.  That  is,  higher  performing 
UAVs  can  substitute  for  lower  numbers. 

•  The  lasso  regression  points  out  the  interaction  between  the  existence  of 
civilian  communication  link  and  the  range  of  this  link.  When  the  civilian 
communication  link  exists,  the  number  of  classified  terrorists  goes  down 
with  increasing  range  of  the  communication  link. 

•  Finally,  the  lasso  regression  indicates  the  interaction  between  the  civilian 
communication  range  with  terrorists  and  time  between  detections  of 
entities  by  UAVs.  The  civilian  communication  range  has  great  effect  on 
decreasing  the  number  of  classified  terrorists  when  the  time  between 
detections  is  long. 

C.  FUTURE  RESEARCH 

This  thesis  provides  many  topics  for  follow-up  studies.  The  following  is  a 
list  of  topics  that  could  be  examined: 

•  Use  different  paths  for  UAVs  to  follow  over  the  area  of  interest. 

•  Use  different  refueling  strategies,  such  as  refueling  on  the  ground  or 
refueling  in  the  air,  to  study  the  effects  of  refueling  time  on  UAV 
classification  performance  in  more  detail.  We  did  not  assume  a  specific 
refueling  strategy  in  our  scenarios. 

•  Conduct  a  more  focused  scenario  analysis  including  only  the  significant 
factors  highlighted  by  this  study. 

•  Examine  the  effects  of  different  terrorist  personalities.  We  assumed  that 
terrorists  have  a  perfect  stealth  value  in  enemy  contact  state  and  their 
propensity  towards  the  destination  degrades  with  the  detection  of  UAVs. 
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APPENDIX  A.  DEPENDENT  VARIABLES  THAT  FORM  THE 
DESIGN  POINTS  WITH  FACTORS  VARIED  IN  THE  DESIGN 


In  addition  to  the  factors  varied  in  the  design,  there  are  also  13  dependent 
input  variables  that  are  set  as  a  function  of  some  factors:  Unmanned  Aerial 
Vehicle  (UAV)  classification  range,  UAV  classification  probability,  time  between 
detections,  and  UAV  fuel  usage  rate.  Here  is  the  list  of  the  dependent  factors  of 
our  study: 


Factor  Name 

Unit 

Type  of 
Variable 

Probability  of  Classification  at 

Med.  Range  in  Default  State 

continuous 

Probability  of  Classification  at 

Min.  Range  in  Default  State 

continuous 

Probability  of  Classification  at 

Med.  Range  in  Enemy  Contact  State 

continuous 

Probability  of  Classification  at 

Min.  Range  in  Enemy  Contact  State 

continuous 

Classification  Range  in  Default 

State  (Med.) 

m 

continuous 

UAV 

Classification  Range  in  Default 

State  (Min.) 

m 

continuous 

Classification  Range  in  Enemy 

Contact  State  (Med.) 

m 

continuous 

Classification  Range  in  Enemy 

Contact  State  (Min.) 

m 

continuous 

Time  Between  Detection  at  Default 
State  (Med.) 

seconds 

continuous 

Time  Between  Detection  at  Default 
State  (Min.) 

seconds 

continuous 

Time  Between  Detection  at  Enemy 
Contact  State  (Med.) 

seconds 

continuous 

Time  Between  Detection  at  Enemy 
Contact  State  (Min.) 

seconds 

continuous 

Fuel  Usage  Rate  in  Refueling  State 

unit 

continuous 

81 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


82 


APPENDIX  B.  SCATTERPLOT  MATRIX  FOR  ALL  FACTORS 


The  scatterplot  matrix  for  all  factors,  including  controllable  and 
uncontrollable  factor  shows  the  space  filling  property  of  our  Nearly  Orthogonal 
Latin  Hypercube  Design  (NOLH). 


Seansrplot  Matrix 
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APPENDIX  C.  DETAILED  COMPARISONS  REPORT  FOR  T-TEST 


The  following  provides  the  paired  t-test  comparisons  for  the  number  of 
classified  Red  Agents  with  different  number  of  UAVs,  with  each  assigned  its  own 
Area  of  Responsibility  (AOR). 


Detailed  Comparisons  Report 

Comparing  2AOR  with  1  AOR 

Difference  8.2998  t  Ratio  2.739749 

Std  Err  Dif  3.0294  DF  512 

Upper  CL  Dif  14.2514  Prob  >  |t|  0.0064* 

LowerCLDif  2.3482  Prob  >t  0.0032* 

Confidence  0.95  Prob  <t  0.9968 

1 

1  '  1 

-10  -5 

1  l  '  l 

5  10 

Comparing  3AOR  with  1  AOR 

Difference  16.4651  t  Ratio  5.4351 

Std  Err  Dif  3.0294  DF  512 

Upper  CL  Dif  22.4167  Prob  >  |t|  <  0001* 

LowerCLDif  10.5135  Prob >t  <0001* 

Confidence  0.95  Prob  <t  1.0000 

1  1  1  1  1  1  1  1  1 

-20  -15  -10  -5  0  5  10  15  20 

Comparing  3AOR  with  2AOR 

Difference  8.1653  t  Ratio  2.695352 

Std  Err  Dif  3.0294  DF  512 

Upper  CL  Dif  14.1169  Prob  >  |t|  0.0073* 

LowerCLDif  2.2137  Prob >t  0.0036* 
Confidence  0.95  Prob  <t  0.9964 

,!  , 

l  1  l  1 

-10  -5 

1  1  1  1 

5  10 

Comparing  4AOR  with  1  AOR 

— i — i — i — |— ^ 

-  1  1  l  '  1 

-20  -10 

10  20 

Difference 

23.1242  t  Ratio 

7.633245 

Std  Err  Dif 

3.0294  DF 

512 

Upper  CL  Dif 

29.0758  Prob  >  |t| 

<0001* 

LowerCLDif 

17.1726  Prob  >t 

<0001* 

Confidence 

0.95  Prob  <t 

1.0000 
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Comparing  4AOR  with  2AOR 


Difference  14.8244  1  Ratio  4.893497 

Std  Err  Dif  3.0294  DF  512 

Upper  CL  Dif  20.7760  Prob  >  |t|  <  0001* 

Lower  CL  Dif  8.8728  Prob  >t  <0001* 

Confidence  0.95  Prob  <t  1.0000 

/ 

/ 

\ 

1  1  1  '  1 

-20-15-10  -5 

5  10  15  20 

Comparing  4AOR  with  3AOR 

Difference  6.6591  t  Ratio  2.198145 

Std  Err  Dif  3.0294  DF  512 

Upper  CL  Dif  12.6107  Prob  >  |t|  0.0284* 

Lower  CL  Dif  0.7075  Prob  >t  0.0142* 

Confidence  0.95  Prob  <t  0.9858 

1 

i — f — i — > — 

-10  -5 

- 1 - 1 — H  i 

5  10 
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APPENDIX  D.  STATISTICAL  RESULTS  OF  THE  MODEL  WITH 

INTERACTIONS 


We  fit  a  saturated  model  first  including  all  pairwise  interactions  among  the 
significant  factors.  Here  are  the  statistical  results  and  sorted  parameter  estimates 
for  the  model  exploring  all  two-way  interactions: 


Response  Transferred_RedAgents 


Actual  by  Predicted  Plot 


Summary  of  Fit 

RSquare  0.93342 

RSquareAdj  0.929302 

Root  Mean  Square  Error  0.408777 

Mean  of  Response  3.315134 

Observations  (orSum  Wgts)  516 


Analysis  of  Variance 


Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

30 

1136.1869 

37.8729 

226.6505 

Error 

485 

81.0426 

0.1671 

Prob  >  F 

C.  Total 

515 

1217.2295 

<  0001* 

Sorted  Parameter  Estimates 


Term 

UAVMaxClassRange_Default 

Red_SensorRange 

UAVPCIassAtMaxRange_EnContact 

NumberofUAVs 

UAVTi  m  e  BtwD  etM  ax_Def  ault 

UAVFuelLevel 

UAVTi  m  e  BtwD  etM  ax_E  n  C  ontad 

(UAVSpeed_Default-270.155)*(Civilian_ComRangeWithRed-5503.88) 
(Red_SensorRange-1 601 .55)*(Civilian_ComRangeWithRed-5503.88) 
(UAVMaxClassRange_Default-5503.88)*(Civilian_ComExisK).50388) 
UAVTimeStepInRefueling 

(UAVPCIassAtMaxRange_EnContact-0.19016)*(UAVTimeBtwDetMax_Default-97.5581) 

(UAVTimeSteplnRefueling-900.465)*(Civilian_ComExist-0.50388) 

Civilian_ComExist 

(NumberofUAVs-2.5)*(UAVMaxClassRange_Default-5503.88) 

(UAVPCIassAtMaxRange_EnContact-0.19016)*(Civilian_ComExisK).50388) 

(UAVSpeed_Default-270.155)*(UAVTimeBtwDetMax_EnContac£67.5581) 

(UAVMaxClassRange_Default-5503.88)*(Red_SensorRange-1601.55) 

UAVSpeed_Default 

(UAVFuelLevel-12153.5)*(UAVTimeSteplnRefueling-900.465) 

(UAVSpeed_Default-270.155)*(UAVTimeSteplnRefueling-900.465) 

(UAVTimeBtwDetMax_Default-97.5581)*(UAVTimeSteplnRefueling-900.465) 

C  i  vi  I  i  a  n_C  o  m  Ra  n  g  e  WithRed 

(UAVMaxClassRange_Default-5503.88)*(UAVTimeBtwDetMax_Default-97.5581) 

(UAVMaxClassRange_Default-5503.88)'(UAVFuelLeveM2153.5) 

(UAVMaxClassRange_Default-5503.88)*(Civilian_ComRangeWithRed-5503.88) 

(NumberofUAVs-2.5)*(Red_SensorRange-1601.55) 

(N  u  m  b  e  rof  U  AVs-2 . 5  )*  (UAVTi  m  e  BtwD  etM  ax_D  ef  a  u  lt-97.5581 ) 

(UAVFuelLevel-12153.5)*(Civilian_ComExisF0.50388) 

(NumberofUAVs-2.5)*(UAVPCIassAtMaxRange_EnContact-0.19016) 


Estimate 

0.0003716 
-0.000776 
5.7639458 
0.4190199 
-0.007961 
2.853e-5 
-0.004241 
5.4167e-7 
8.114e-8 
8.6867e-5 
-0.000333 
-0.035473 
-0.000669 
-0.193103 
0.0000319 
-2.161187 
-0.000025 
3.9013e-8 
-0.000755 
5.3982e-8 
2.4764e-6 
4.7689e-6 
-2.525e-5 
-5.43 1e-7 
-4.297e-9 
-8.966e-9 
-5.325e-5 
-0.000857 
1.5637e-5 
0.4014043 


Std  Error 

t  Ratio 

6.801e-6 

54.64 

0.000022 

-35.14 

0.220964 

26.09 

0.016096 

26.03 

0.000373 

-21.34 

3.259e-6 

8.75 

0.000577 

-7.35 

7.813e-8 

6.93 

1.175e-8 

6.91 

0.000014 

6.17 

5.74e-5 

-5.80 

0.006614 

-5.36 

0.000126 

-5.33 

0.036357 

-5.31 

6.057e-6 

5.27 

0.454446 

-4.76 

5.3e-6 

-4.70 

8.565e-9 

4.55 

0.000172 

-4.39 

1.254e-8 

4.30 

5.858e-7 

4.23 

1.183e-6 

4.03 

6.779e-6 

-3.72 

1.532e-7 

-3.54 

1.399e-9 

-3.07 

3.275e-9 

-2.74 

1.974e-5 

-2.70 

0.000333 

-2.57 

0.000007 

2.23 

0.19737 

2.03 

Prob>|t| 

<0001* 
<0001* 
<.0001* 
<.0001* 
<.0001* 
<  0001* 
<0001* 
<.0001* 
<.0001* 
<.0001* 
<0001* 
<0001* 
<.0001* 
<.0001* 
<.0001* 
<.0001* 
<0001* 
<.0001* 
<.0001* 
<0001* 
<.0001* 
<0001* 
0.0002* 
0.0004* 
0.0022* 
0.0064* 
0.0072* 
0.0104* 
0.0260* 
0.0425* 
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APPENDIX  E.  TREE  STRUCTURE  FOR  FIRST  13  SPLITS 


Regression  trees  recursively  partition  data  to  find  an  optimal  split  on  the 
response.  After  each  split,  the  succeeding  ideal  split  is  determined  within  each 
partition.  We  grow  a  regression  tree  with  20  splits.  The  tree  structure  for  first  13 
splits  is  provided  below. 
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